UQ and Trustworthy AI Algorithms for Complex Systems and Social Good

Prediction of storm-surge hazard and impacts within planning (pre-disaster), emergency management and post-disaster settings has emerged as a key priority in natural hazard risk mitigation efforts. Migration towards coasts as well as concerns related to the future effects of climate change, further stress the importance of research efforts that attempt to address this priority. Numerical advances in storm surge prediction is one of the more important such efforts. These advances have produced high-fidelity simulation models that permit a detailed representation of hydrodynamic processes and therefore support high-accuracy forecasting. Unfortunately, the computational burden of such numerical models is large, requiring thousands of CPU hours for each simulation, something that limits their applicability for hurricane risk assessment. This prohibits their broader use in regional planning or emergency response management efforts. This seminar will examine how machine learning advances have been recently promoted to address this challenge, and how the integration of such techniques can be established to better serve the needs of the relevant decision makers (planners, emergency managers). Some emphasis will be initially placed on technical aspects for integrating surrogate modeling techniques to provide surge predictions using a database of high-fidelity, synthetic storms, with the goal of maintaining the accuracy of the numerical model utilized to produce this database, while providing greatly enhanced computational efficiency. This ultimately supports great versatility in leveraging high-fidelity modeling to support regional flood studies (supported by FEMA or Army Corps of Engineers) and real-time emergency response management (supported by NOAA). The integration of dimensionality reduction techniques, multi-fidelity approaches and graph-based neural network predictions are discussed in this context. The discussion then moves on to examining how these developments and the advantages they can offer need to be promoted in order to gain acceptance by the community they intend to serve (aforementioned agencies and also planner and emergency response managers), corresponding in this case to non-technical end-users with respect to the underlying computational statistics methods. This is an essential step in the interaction between Computer Science (CS) and the people/organizations it should assist, and needs careful planning in order to promote broad adoption and actual empowerment of the end-users. Discussion focuses on the feedback loop that needs to be established in this interaction in order to achieve the desired outcome.

Understanding and predicting weather and climate extremes pose significant challenges due to the complex interactions among atmospheric variables, nonlinearity of underlying processes, and scarcity of data. Flexible statistical and machine learning approaches often fail to capture these complexities or lack interpretability, hindering practical applications. This talk will explore the evolution of interpretable machine learning methodologies applied to this domain, beginning with foundational work from the late 1990s, which introduced methods for visualizing functions computed by neural networks. The talk will trace subsequent advancements and applications, demonstrating how simple constraints on model architecture and parameters can lead to more interpretable and transparent models. Applications to predicting and analyzing weather and climate extremes — ranging from operational air quality forecasting, analysis of streamflow extremes, and projections of time-of-exceedance of global warming levels — will be presented, with an emphasis on how interpretability helped offer actionable insights to stakeholders and policy makers.

Operator models are regression algorithms between Banach spaces of functions and have become a key tool for learning large scale dynamical systems. Recent advances in deep neural operators have dramatically improved the accuracy and scalability of operator modeling, but lack an inherent notion of predictive uncertainty. We introduce local spectral conformal inference (LSCI), a new adaptive conformal inference procedure for black box operator models that localizes, or adapts, the spectra of the prediction sets to the spectra of the predictions. LSCI prediction sets are locally adaptive, risk controlling, and have approximate conditional coverage. We evaluate LSCI against baseline conformal methods on Navier-Stokes simulations and large weather forecasting tasks.

Land in the UK is a scarce resource. To meet the negative emissions required to offset aviation and agriculture for Net Zero, we must change the way we use the land. But what options should be taken where? Should we plant trees? restore peatland? invest in bioenergy crops or other forms or renewables? Move to agroforestry? If we could answer those questions across the land, given that we don’t own the land, how should we incentivise those that do to provide the change we need? In this talk we focus on one set of decisions, where and when to plant trees. Even here, we find an astronomical number of options impacting a host of ecosystem services beyond CO2 sequestration. I will present a framework that harnesses UQ to move beyond scenario analysis (a detailed exploration of 3 or 4 planting options) whilst avoiding disempowering decision makers by using relying solely on optimisation. Our approach is to attempt to invert the decision problem and to offer users a guided tour of their decision space: finding a space of options compatible with user-adjustable targets on ecosystem service levels, and exploring the diversity of options and tradeoffs within that space.

We propose a framework to perform probabilistic data assimilation  (DA) using conditional score-based diffusion models. Conditional score-based diffusion models are generative models that learn to approximate the score function of a conditional distribution using samples from the joint distribution. We apply this approach to approximate the update step of the Bayes filter and arrive at a particle-based DA approach. The application of this approach only requires simulating the forward model. Hence, the proposed approach can accommodate black-box forward dynamics models and complex measurement noise. We demonstrate the efficacy of the proposed approach on canonical and challenging DA problems. The results show that the proposed framework can solve large-scale physics-based DA problems efficiently.

Over more than two decades, research on simulation-based design under uncertainty has promoted the rigorous use of statistical analysis (data science) and probabilistic theory in engineering design. Novel algorithmic approaches have been developed to accelerate computational design innovation and optimization, and enable knowledge discovery in simulation-based design characterized by expensive multi-physics and multiscale simulations, high dimensionality, multi-modal data sources, and various sources of uncertainty.  Recent years have seen a significant growth of using these methods for design of advanced materials systems. In this talk, we will introduce the success of using simulation-based design under uncertainty for designing both heterogeneous nano- and microstructural materials and architectural metamaterials, by integrating knowledge and representation from multiple disciplines and domains such as materials, manufacturing, structural mechanics, and design optimization. Challenges in design representation, design evaluation, and design synthesis associated with material systems will be introduced together with new techniques of microstructure characterization and reconstruction, multiscale uncertainty quantification and propagation, mixed-variable Gaussian process modeling, Bayesian optimization, and data-driven multiscale topology optimization.

"The increasing power of computing platforms and the recent advances in data science techniques have fostered the development of data-driven computational models of engineering systems with considerably improved prediction accuracies. An important feature of these modeling approaches is the reliance on data to develop reduced-order models of physical phenomena involved and/or the characterization of the uncertainty associated with the models or their parameters.  In the latter case, the quantification of the impact of such uncertainty on the quantities of interest is key to assess the validity of a given model or perform reliability analysis, among other tasks. However, for complex engineering systems, such as those featuring multi-physics and multi-scale phenomena, simulation models are computationally expensive. These, in turn, pose significant challenges on standard UQ and reliability estimation approaches.   I will start this talk with a brief discussion on the challenges associated with uncertainty quantification (UQ) of complex systems and a high-level introduction to recent work performed by the UQ and Data-driven Modeling group at CU Boulder to tackle these challenges. I will focus on model reduction approaches (linear and non-linear) for efficient UQ, generative modeling, and failure probability estimation. I will present application examples to highlight the efficiency of these multi-fidelity approaches and their wide applicability to a broad range of problems."

We present a probabilistic framework to represent and quantify model-form uncertainties in the reduced-order modeling of complex systems using operator inference techniques. Such uncertainties can arise in the selection of an appropriate state and can dramatically impact the robustness of the predictions. The proposed method captures these uncertainties by expanding the approximation space through the randomization of the projection matrix. The efficacy of the approach is assessed on canonical problems in fluid mechanics.

We will re-examine two popular use-cases of Bayesian approaches: model selection, and robustness to distribution shifts. The marginal likelihood (Bayesian evidence) provides a distinctive approach to resolving foundational scientific questions --- "how can we choose between models that are entirely consistent with any data?" and "how can we learn hyperparameters or correct ground truth constraints, such as intrinsic dimensionalities, or symmetries, if our training loss doesn't select for them?". There are compelling arguments that the marginal likelihood automatically encodes Occam's razor. There are also widespread practical applications, including the variational ELBO for hyperparameter learning. However, we will discuss how the marginal likelihood is answering a fundamentally different question than "will my trained model provide good generalization?". We consider the discrepancies and their significant practical implications in detail, as well as possible resolutions. Moreover, it is often thought that Bayesian methods, representing epistemic uncertainty, ought to have more reasonable predictive distributions under covariate shift, since these points will be far from our data manifold. However, we were surprised to find that high quality approximate Bayesian inference often leads to significantly decreased generalization performance. To understand these findings, we investigate fundamentally why Bayesian model averaging can deteriorate predictive performance under distribution and covariate shifts, and provide several remedies based on this understanding.

In this talk, I will summarize the recent report on AI Engineering which was developed under the auspices of NSF funded project, Engineering Research Visioning Alliance. After this, I will highlight some key areas where uncertainty quantification can make important contributions.