Reliability & Risk Management Lab

For two probability measures $\rho$ and $\pi$ on $[−1,1]^\mathbb{N}$ we investigate the approximation of the triangular Knothe-Rosenblatt transport $T: [−1,1]^\mathbb{N} \to [−1,1]^\mathbb{N}$ that pushes forward $\rho$ to $\pi$. Under suitable assumptions, we show that $T$ can be approximated by rational functions without suffering from the curse of dimension. Our results are applicable to posterior measures arising in certain inference problems where the unknown belongs to an (infinite-dimensional) Banach space. In particular, we show that it is possible to efficiently approximately sample from certain high-dimensional measures by transforming a lower-dimensional latent variable.

@article{zech-infdim-2021,
  author     = "J. Zech and Y. Marzouk",
  journal    = "Constructive Approximation",
  title      = "Sparse approximation of triangular transports. Part II: the infinite dimensional case",
  volume     = "in press",
  year       = "2021",
  abstract   = "For two probability measures $\rho$ and $\pi$ on $[−1,1]^\mathbb{N}$ we investigate the approximation of the triangular Knothe-Rosenblatt transport $T: [−1,1]^\mathbb{N} \to [−1,1]^\mathbb{N}$ that pushes forward $\rho$ to $\pi$. Under suitable assumptions, we show that $T$ can be approximated by rational functions without suffering from the curse of dimension. Our results are applicable to posterior measures arising in certain inference problems where the unknown belongs to an (infinite-dimensional) Banach space. In particular, we show that it is possible to efficiently approximately sample from certain high-dimensional measures by transforming a lower-dimensional latent variable."
}

Many Bayesian inference problems involve target distributions whose density functions are computationally expensive to evaluate. Replacing the target density with a local approximation based on a small number of carefully chosen density evaluations can significantly reduce the computational expense of Markov chain Monte Carlo (MCMC) sampling. Moreover, continual refinement of the local approximation can guarantee asymptotically exact sampling. We devise a new strategy for balancing the decay rate of the bias due to the approximation with that of the MCMC variance. We prove that the error of the resulting local approximation MCMC (LA-MCMC) algorithm decays at roughly the expected $1/\sqrt{T}$ rate, and we demonstrate this rate numerically. We also introduce an algorithmic parameter that guarantees convergence given very weak tail bounds, significantly strengthening previous convergence results. Finally, we apply LA-MCMC to a computationally intensive Bayesian inverse problem arising in groundwater hydrology.

@article{davis-lamcmc-2020,
  author     = "A. Davis and Y. M. Marzouk and N. Pillai and A. Smith",
  journal    = "Preprint",
  title      = "Rate-optimal refinement strategies for local approximation MCMC",
  year       = "2021",
  abstract   = "Many Bayesian inference problems involve target distributions whose density functions are computationally expensive to evaluate. Replacing the target density with a local approximation based on a small number of carefully chosen density evaluations can significantly reduce the computational expense of Markov chain Monte Carlo (MCMC) sampling. Moreover, continual refinement of the local approximation can guarantee asymptotically exact sampling. We devise a new strategy for balancing the decay rate of the bias due to the approximation with that of the MCMC variance. We prove that the error of the resulting local approximation MCMC (LA-MCMC) algorithm decays at roughly the expected $1/\sqrt{T}$ rate, and we demonstrate this rate numerically. We also introduce an algorithmic parameter that guarantees convergence given very weak tail bounds, significantly strengthening previous convergence results. Finally, we apply LA-MCMC to a computationally intensive Bayesian inverse problem arising in groundwater hydrology.",
  keywords   = "Markov chain Monte Carlo, local regression, Bayesian inference, surrogate models, sampling"
}

We propose a framework for solving high-dimensional Bayesian inference problems using structure-exploiting low-dimensional transport maps or flows. These maps are confined to a low-dimensional subspace (hence, lazy), and the subspace is identified by minimizing an upper bound on the Kullback–Leibler divergence (hence, structured). Our framework provides a principled way of identifying and exploiting low-dimensional structure in an inference problem. It focuses the expressiveness of a transport map along the directions of most significant discrepancy from the posterior, and can be used to build deep compositions of lazy maps, where low-dimensional projections of the parameters are iteratively transformed to match the posterior. We prove weak convergence of the generated sequence of distributions to the posterior, and we demonstrate the benefits of the framework on challenging inference problems in machine learning and differential equations, using inverse autoregressive flows and polynomial maps as examples of the underlying density estimators.

@article{brennanbigoni-lazy-2020,
  author     = "D. Bigoni and O. Zahm and A. Spantini and Y. M. Marzouk",
  journal    = "Advances in Neural Information Processing Systems (NeurIPS)",
  title      = "Greedy inference with structure-exploiting lazy maps",
  year       = "2020",
  abstract   = "We propose a framework for solving high-dimensional Bayesian inference problems using structure-exploiting low-dimensional transport maps or flows. These maps are confined to a low-dimensional subspace (hence, lazy), and the subspace is identified by minimizing an upper bound on the Kullback--Leibler divergence (hence, structured). Our framework provides a principled way of identifying and exploiting low-dimensional structure in an inference problem. It focuses the expressiveness of a transport map along the directions of most significant discrepancy from the posterior, and can be used to build deep compositions of lazy maps, where low-dimensional projections of the parameters are iteratively transformed to match the posterior. We prove weak convergence of the generated sequence of distributions to the posterior, and we demonstrate the benefits of the framework on challenging inference problems in machine learning and differential equations, using inverse autoregressive flows and polynomial maps as examples of the underlying density estimators.",
  keywords   = "Bayesian inference, greedy approximation, transport"
}

Multivariate functions encountered in high-dimensional uncertainty quantification problems often vary along a few dominant directions in the input parameter space. We propose a gradient-based method for detecting these directions and using them to construct {ridge approximations} of such functions, in a setting where the functions are vector-valued (e.g., taking values in $\mathbb{R}^n$). The methodology consists of minimizing an upper bound on the approximation error, obtained by {subspace Poincaré inequalities}. We provide a thorough mathematical analysis in the case where the parameter space is equipped with a Gaussian probability measure. The resulting method generalizes the notion of active subspaces associated with scalar-valued functions. A numerical illustration shows that using gradients of the function yields effective dimension reduction. We also show how the choice of norm on the codomain of the function has an impact on the function’s low-dimensional approximation.

@article{zahm-dimred-2020,
  author     = "O. Zahm and P. Constantine and C. Prieur and Y. M. Marzouk",
  journal    = "SIAM Journal on Scientific Computing",
  number     = "1",
  pages      = "A534-A558",
  title      = "Gradient-based dimension reduction of multivariate vector-valued functions",
  volume     = "42",
  year       = "2020",
  abstract   = "Multivariate functions encountered in high-dimensional uncertainty quantification problems often vary along a few dominant directions in the input parameter space. We propose a gradient-based method for detecting these directions and using them to construct {ridge approximations} of such functions, in a setting where the functions are vector-valued (e.g., taking values in $\mathbb{R}^n$). The methodology consists of minimizing an upper bound on the approximation error, obtained by {subspace Poincaré inequalities}. We provide a thorough mathematical analysis in the case where the parameter space is equipped with a Gaussian probability measure. The resulting method generalizes the notion of active subspaces associated with scalar-valued functions. A numerical illustration shows that using gradients of the function yields effective dimension reduction. We also show how the choice of norm on the codomain of the function has an impact on the function's low-dimensional approximation.",
  keywords   = "high-dimensional function approximation, dimension reduction, active subspace, ridge approximation, Karhunen-Loève decomposition, Poincaré inequality"
}