As I go around presenting the idea of probabilistic numerics to various audiences, certain questions about related areas come up repeatedly. This post explains how probabilistic numerics compares to the established area of Uncertainty Quantification. Subsequent posts will discuss connections to certain kinds of stochastic numerical methods, and the young and popular area of Bayesian Optimization.
A disclaimer: Obviously, everyone has different opinions about the scope and purpose of certain concepts and academic fields. And I am not an expert in the areas discussed here. This post relates a part of my own personal justification, why I think certain ideas are novel and interesting. It is not intended as a holistic overview of a field. If anyone disagrees with characterizations I make here, I would be glad if you could relate your opinion in the comments section below. I’m grateful to Mike Osborne for some comments on a draft for this post.
Uncertainty Quantification (UQ) is a relatively young area (but considerably older than probabilistic numerics) at the boundary of numerical mathematics and statistics, dealing with, as SIAM and the ASA define it: “the interface of complex modeling of processes and data, especially characterizations of the uncertainties inherent in the use of such models.” This description is (probably deliberately) vague and might well describe all of statistics. As Tony O’Hagan writes (O’Hagan, 2013):
It is the AM [applied mathematics] community that is responsible for the term UQ (Uncertainty Quantification). To a member of the Stat community, however, UQ seems a curiously inappropriate term because (a) it sounds as though it should cover the whole of Statistics, since one way of viewing the science of Statistics is precisely as the science of quantifying uncertainty, whereas (b) the uncertainties that are studied in the AM community do not even cover the range of uncertainties that statisticians would recognise in the predictions of models. Nevertheless, UQ is the de facto term that has been accepted by the SIAM/ASA joint initiative.
In my impression – and my knowledge of the field is very limited – UQ is still evolving and continues to expand and address new questions. But the principle problem at its core is the propagation of input uncertainty. In very simple terms: Consider a complicated model mapping inputs to outputs . If I “wiggle” at a given value according to some perturbation , what is the effect on ? For example, may be a climate model, or a flow field over some complicated airframe.
There is a variety of methods used for this task in UQ, including the popular concept of a polynomial chaos expansion, which, intriguingly, has a connection to the Gaussian process surrogates used in Bayesian Optimization, which will be the subject of a later blog post on connections. And clearly, the uncertainty propagation described above addresses a certain type of probabilistic uncertainty we are also concerned with in Probabilistic Numerics: Given uncertain inputs to a problem and a particular algorithm used to solve it, what should the uncertainty over the output be?
Nevertheless, our questions in probabilistic numerics take a different, and in some cases broader view that warrants its own research. For us, the object of interest is the computation itself, rather than its reaction to a change in input. Questions we would like to answer in probabilistic numerics include, for a particular computation task: Is it performed as efficiently as possible (from an information theoretic perspective) by a particular algorithm? What kind of information does it collect about related objects? And could it be made more adaptive to salient prior knowledge about a certain task at hand?
To get an intuition for the many different ways that uncertainty can play a role in computation, let’s look at an elementary problem: the linear problem of finding such that with and a matrix . Here are some questions one could ask:
Uncertainty from ill-posedness: If , there are generally many that solve the problem. What is the space of such admissible ? This is (a base case of) the kind of uncertainty typically studied in statistics, when a dataset is not sufficiently informative to identify the parameters of a model.
Uncertainty from imprecise inputs: If is perturbed to , what is the effect on ? This is the elementary version of a central point of interest for Uncertainty Quantification. (But, as Tony O’Hagan says above, this kind of uncertainty is of course also of interest in statistics sometimes).
These two questions are classic ones, and in this linear problem, they have straightforward, or in fact trivial answers, while in nonlinear problems, they can pose formidable challenges. With probabilistic numerics, we add another set of questions:
To summarize: Probabilistic numerics focusses on the computations used to solve a particular problem: what is the uncertainty added by performing the computation approximately, and how can this information be used in ways not yet anticipated? PN and UQ can live very well next to each other. PN can learn much from UQ, in particular in areas like uncertainty propagation. Conversely, I hope colleagues working in UQ will be interested in how our results on PN can help in large scale UQ. The existence of UQ as such is no reason not to study PN.
This paper proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems Bx = b with positive definite B for x. The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of B, which can be used to estimate errors. Recent probabilistic interpretations of the secant family of quasi-Newton optimization algorithms are extended. Combined with properties of the conjugate gradient algorithm, this leads to uncertainty-calibrated methods with very limited cost overhead over conjugate gradients, a self-contained novel interpretation of the quasi-Newton and conjugate gradient algorithms, and a foundation for new nonlinear optimization methods.
@article{2014arXiv14022058H, author = {{Hennig}, P.}, journal = {SIAM J on Optimization}, month = jan, title = {{Probabilistic Interpretation of Linear Solvers}}, year = {2015}, link = {http://epubs.siam.org/doi/abs/10.1137/140955501?journalCode=sjope8}, volume = {25}, issue = {1}, file = {http://probabilistic-numerics.org/assets/pdf/HennigLinear2015.pdf} }
@techreport{ohagan13-polyn-chaos, author = {O'Hagan, Anthony}, title = {Polynomial Chaos: A Tutorial and Critique from a Statistician's Perspective}, institution = {University of Sheffield, UK}, year = {2013}, month = may }