Machine Learning for Climate and Weather Applications

Effective and efficient data assimilation is challenging for complex turbulent systems with partial observations. In this work, we propose a new hybrid nonlinear data assimilation framework for such systems. This hybrid framework contains four steps: (1). With the observed state variable data, we build a stochastic surrogate model using stochastic parameterization Kalman filters (SPEKF); (2). We train a set of long short-term memory (LSTM) neural networks (NN) between observed and unobserved variables; (3). We apply the ensemble transform Kalman filter (ETKF) to obtain the posterior mean of observed variables and use the filtered data as inputs in LSTM-NN from step (2) to recover the unobserved variables; and (4). We quantify the uncertainty for the recovered unobserved state variables using the efficient conditional Gaussian mixture filter. One application we considered is the recently proposed precipitation quasi-geostrophic (PQG) equations. Compared with classic quasi-geostrophic equations, PQG introduces an additional moisture variable and Heaviside nonlinearities to account for phase changes of water and thus is more complex in physics and expensive in computation. The numerical results show that compared with bare ETKF or localized (ETKF) (LETKF) scenarios, the new hybrid framework is more efficient and accurate in reproducing the dynamics and statistical features for both observed and unobserved state variables.

Earth’s climate is chaotic and noisy. Finding usable signals amidst all of the noise can be challenging. Here, I will demonstrate how explainable artificial intelligence (XAI) techniques can sift through vast amounts of climate data and push the bounds of scientific discovery. But machine learning models are only as capable as the scientists designing them, and climate science requires the crafting of domain specific XAI methods, both to gauge the trustworthiness of the XAI’s predictions and quantify uncertainty, but also to uncover predictable signals we didn’t know were there.

Global climate and earth system models (ESMs), which numerically solve partial differential equations with high performance simulations, continue to have knowledge gaps and exhibit intrinsic variability for stakeholder relevant variables and resolutions. Data-driven sciences integrated with process understanding, especially the physics or biogeochemistry that may not be fully captured within the simulations, are critical to improve model parameterizations, develop a comprehensive characterization of variability and uncertainty, and extract scientific insights from archived model simulations. Furthermore, data-driven discrete event simulations have been proposed to incorporate societal dimensions such as management of watersheds in the land component of earth system models. The first part of this presentation will rely on our work at the Sustainability and Data Sciences Laboratory (SDS Lab) and the extant literature to elucidate the role of Artificial Intelligence (AI) and high performance computing (HPC), along with falsifiability and Uncertainty Quantification (UQ), in three areas, specifically, post-processing ESM simulations with knowledge-guided AI for extracting stakeholder and policy relevant insights, embedding AI within ESM for improving processes and parameterizations, and incorporating human and societal dimensions within ESMs. The second part of the presentation will focus on Machine Learning (ML), even touching upon the “unreasonable effectiveness” of Deep Learning, based downscaling in climate with a particular focus on UQ along with evaluation and falsifiability, such that ESM simulations at lower resolutions can be credibly translated to information across local to regional scales to enable stakeholder decisions and policy. The presentation will conclude with a short discussion on making climate science actionable by relying not just on governmental or intergovernmental action but also through innovations in the private sector via large corporations and sustainable startups.

The forecast covariance matrix is a crucial component of most operational data assimilation methods, and ensemble forecasts are frequently used to help estimate the forecast covariance. In the context of coupled Earth system models or high-resolution climate model components, ensemble forecasts can be extremely computationally expensive. Developing machine learning methods to reduce the cost of an ensemble forecast is an active area of research. The approach taken here relies on a single forecast (or a few) using the expensive physics-based model, and uses machine learning to generate an ensemble of synthetic model states that are similar to (analogs of) the single physics-based forecast. Specifically, autoencoders, including variational, are trained to reconstruct model states from a training set. Once trained, the ensemble is generated by encoding the single forecast into the latent space, then adding noise in latent space, then decoding the ensemble back to the model space. For very high-dimensional models, a single model state can be constructed from local ‘patches’ stitched together. The performance of data assimilation using these (and related) analog ensembles is explored in an idealized Lorenz model and in a high-resolution coupled quasigeostrohic model.

Climate models depend on dynamics across many spatial and temporal scales. It is infeasible to resolve all of these scales. Instead, the physics at the smallest scales is represented by parameterization schemes that link what is unresolvable to variables resolved on the grid scale. A large source of uncertainty in climate predictions comes from the calibration of empirical parameters in such parameterization schemes, and these uncertainties are generally not quantified. The uncertainties can be reduced and quantified with data that may have limited availability in space and time, for example, data from field campaigns or from targeted high-resolution simulations in limited areas. But the sensitivity of simulated climate statistics, such as precipitation rates, to parameterizations varies in space and time, raising the question of where and when to acquire additional data so as to optimize the information gain from the data. We construct an automated algorithm that finds optimal regions and time periods for such data acquisition, to maximize the information the data provides about uncertain parameters. We use a Bayesian framework to characterize the uncertainty and apply the new Calibrate-Emulate-Sample (CES) methodology to accelerate its calculation. The combined algorithm is parallelizable, and is efficient with respect to evaluations of the model. In proof-of-concept simulations with an idealized global atmosphere model, we show that our algorithm can successfully identify informative regions and times, even in cases where physics-based intuition may lead to sub-optimal choices.

Modern ideas coming from dynamical systems theory and statistical mechanics allow for a much-improved understanding of the relationship between climate variability and climate response to to forcings. This is also leading to the possibility of finding a synthesis between Hasselmann and Lorenz’s visions of climate, the first based on stochasticity, the latter rooted in deterministic chaos. I will focus on the global stability properties of the climate, and specifically on the dichotomy between the co-existing warm and snowball states, which has had a key importance for the development of life. When stochastic forcing is included, one observes transitions between the competing basins of attraction. For weak Gaussian noise laws, large deviation laws define the invariant measure, the statistics of escape times, and typical escape paths called instantons. The Melancholia state, the chaotic saddle embedded in the boundary between the basins of attraction, is the gateway for noise-induced transitions. The metastability can be understood in terms of an energy-like landscape with valleys and mountain ridges defined by the Graham’s quasipotential. Finally, I will discuss generalizations of these results due to the presence of a larger number of competing states and on the consideration of other classes of noise laws and will discuss how data driven methods can help identifying non-trivial metastable states. Refs:Lucarini, T. Bodai, Transitions across Melancholia States in a Climate Model: Reconciling the Deterministic and Stochastic Points of View, Phys. Rev. Lett. 122,158701 (2019)Lucarini, T. Bodai, Global Stability Properties of the Climate: Melancholia States, Invariant Measures, and Phase Transitions, Nonlinearity 33, R59 (2020)Margazoglou, T. Grafke, A. Laio, V. Lucarini, Dynamical Landscape and Multistability of the Earth’s Climate, Proc. R. Soc. A 477, 2021001920210019 (2021)Lucarini, L. Serdukova, L., and G. Margazoglou, Lévy-noise versus Gaussian-noise-induced Transitions in the Ghil-Sellers Energy Balance Model, Nonlin. Processes Geophys. https://doi.org/10.5194/npg-2021-34 (2022)Ragon, V. Lembo, V. Lucarini, C. Verard, J, Kasparian, M. Brunetti, Robustness of competing climatic states, J. Climate 35, 2769-2784 (2022)

Understanding extreme events and their probability is key for the study of climate change impacts, risk assessment, adaptation, and the protection of living beings. Extreme heatwaves are, and likely will be in the future, among the deadliest weather events. Forecasting their occurrence probability a few days, weeks, or months in advance is a primary challenge for risk assessment and attribution, but also for fundamental studies about processes, dataset and model validation, and climate change studies.  We will demonstrate that deep neural networks can predict the probability of occurrence of long lasting 14-day heatwaves over France, up to 15 days ahead of time for fast dynamical drivers (500 hPa geopotential height fields), and at much longer lead times for slow physical drivers (soil moisture). This forecast is made seamlessly in time and space, for fast hemispheric and slow local drivers.  A key scientific message is that training deep neural networks for predicting extreme heatwaves occurs in a regime of drastic lack of data. We suggest that this is likely the case for most other applications of machine learning to large scale atmosphere and climate phenomena. We discuss perspectives for dealing with this lack of data issue, for instance using rare event simulations.  Rare event simulations are a very efficient tool to oversample drastically the statistics of rare events. We will discuss the coupling of machine learning approaches, for instance the analogue method, with rare event simulations, and discuss their efficiency and their future interest for climate simulations.

With the goal of developing a data-driven parameterization of gravity waves (GWP) for use in general circulation models, we train various machine learning architectures to emulate Alexander-Dunkerton 1999, an existing GWP scheme. We diagnose the disparity between online and offline performance of the trained emulators by identifying a subspace of the phase space that is prone to large errors and sparse samples, and develop a sampling algorithm to treat biases that stem from underrepresentation. This strategy can be used for regression tasks over long-tailed (and other imbalanced) distributions. We find that error-prone samples often have larges shears in the wind profile– this is corroborated with physical intuition as large shears indicate many breaking levels, which requires a more complex, nonlocal computation. To remedy this, we develop a sampling strategy that performs a parameterized histogram equalization.  The sampling algorithm uses a linear mapping from the original histogram to the uniform histogram parameterized by $t in [0,1]$. Parameters $t$ and “maximum repeat” assign each bin a new probability. The new probability is applied in two different implementations: 1) by sampling the bins to adjust the training set distribution; 2) by weighting the loss function to achieve the same effect in expectation. We find that this strategy improves the errors at the tail of the distribution except at the extreme end, while maintaining minimal loss of accuracy at the peak of the distribution.

NVIDIA is committed to helping address climate change. Recently our CEO announced the Earth-2 initiative, which aims to build digital twins of the Earth and a dedicated supercomputer, E-2, to power them. Two central goals of this initiative are: (i) Computational: Enable high-resolution climate predictions with credible cloud physics; and (ii) Societal: Nimbly serve interactive, useful, next-generation climate predictions via NVIDIA Omniverse. These predictions will help plan for the disastrous impacts of climate change well in advance and develop strategies to mitigate and adapt to change.

Next-generation km-scale climate simulations are prohibitively expensive and produce unmanageable data volumes. Therefore, the above-mentioned goals depend on achieving orders-of-magnitude speedup and data compression via “tethering” a skillful ML surrogate to checkpoints of km-scale accelerated hybrid ML-climate models.

In this context, we present results from FourCastNet, a sophisticated, transformer-based adaptive Fourier Neural Operator (FNO) deep learning model for auto-regressive forecasting. While ultimately intended for use in climate, its architecture and predictive skill limits are being refined in the context of global high-resolution weather prediction.

We conclude with a roadmap of Earth-2 that encompasses weather and climate goals and outlines engineering innovations required for the breakthroughs that building digital twins of the Earth demands.

I will discuss two recent publications that are relevant for automated learning of climate and weather phenomena. The first of these is concerned with finding large-scale approximate cycles in climate data. These ideas are illustrated by finding the dominant cyclic behaviour in the Lorenz flow, and by producing an improved characterisation of the El-Nino Southern Oscillation from spatiotemporal data. The second part of the discussion addresses the data-driven identification of coherent regions in the phase space of a dynamical system. In particular, I will discuss the difficult, but common, situation where there are several regime changes in the dynamics that cause the emergence or destruction of coherence. Each of these aspects, cyclicity and coherence, contributes to learning a richer time-varying view of weather and climate dynamics.

Data-driven methods such as machine learning may hold the potential to dramatically speed up climate simulation, enabling more precise estimation of statistics such as return periods of extreme weather events. But it is questionable whether such models can be trusted to generalize into the far tails of the climatic probability distribution, beyond the training data, where the human and ecological impacts are greatest. High-resolution, physics-based weather models remain the most reliable tools to simulate extreme events, but for computational reasons are typically only run for several weeks at a time—not nearly enough to estimate climatic-timescale return periods. In this talk, I will describe a collection of simple but statistically sound methods to estimate long return times from short weather forecasts. With proper weighting of different trajectories, we can stitch together many partial trajectories into longer ones, which are not independent but come from the same distribution. We illustrate on a dramatic weather phenomenon, sudden stratospheric warming (SSW), which is intermittent and scarcely predictable but very impactful on midlatitude winter weather regimes. Several different approaches, including Markov state modeling and linear inverse modeling, indicate similar return periods. Extracting all available information from existing weather models is an important first step to generate null hypotheses and identify reaction pathways to pave the way for event-focused design of ensemble simulations.

Global climate models represent small-scale processes, such as clouds and convection, using subgrid models known as parameterizations. Traditional parameterizations are usually based on simplified physical models, and inaccuracies in these parameterizations are a main cause for the large uncertainty in climate projections. One alternative to traditional parameterizations is to use machine learning to learn new parameterizations which are data driven. Attempts to use machine learning for developing new parameterizations for the atmosphere have mainly focused on parameterizing the effects of subgrid processes on thermodynamic and moisture variables, but subgrid momentum transport is also important in simulations of the atmospheric circulation. In this study we use neural networks to develop a parameterization of subgrid momentum transport that learns from coarse-grained output of a high-resolution atmospheric simulation in an idealized aquaplanet domain. We show that substantial subgrid momentum transport occurs due to convection and non-orographic gravity waves. The neural network parameterization we develop has a structure that ensures the conservation of momentum, and it has reasonable skill in predicting momentum fluxes associated with convection. However, the neural network is less accurate for momentum than for energy and moisture, possibly due to the challenging task of predicting momentum fluxes associated with gravity waves, and the difficulty in predicting the sign of momentum fluxes during convection. When this parameterization is implemented in an atmospheric model at coarse resolution it leads to stable simulations, and some characteristics of the atmospheric circulation improve, while some characteristics are sensitive to the exact model configuration. To investigate how the accuracy of neural network momentum parameterization can be improved, we train neural networks using non-local inputs spanning over 3×3 columns of inputs. We find that including the non-local inputs substantially improves the prediction of subgrid momentum transport compared to a single-column formulation. This shows that a single-column approach might not be ideal for the parameterization problem since certain atmospheric phenomena, such as organized convective systems, can cross multiple grid boxes. Overall, our results show that neural-networks parameterization of momentum transport have the potential to improve the representation of subgrid momentum transport, and they also highlight the difficulty in parameterizing subgrid momentum transport.

Geophysical datasets are characterized by unique challenges: multiple scales in space and time, strong nonlinear coupling of dynamical components, and a large number of positive Lyapunov exponents, i.e. instabilities. These properties make the prediction and uncertainty quantification in geophysical settings a problem of unique complexity. Contemporary ocean, atmospheric and climate models aim to overcome these challenges by accurate numerical discretization of the governing equations, careful parameterizations, and/or data-assimilation schemes. However, the resulted models are typically very complex and expensive, making the quantification of rare events over large time horizons a formidable task. This is especially important for the situation where one is interested to evaluate the effectiveness of a particular policy or measure. This work aims to overcome some of these limitations and consists of two parts. In the first part our scope is to build a non-intrusive correction-operator that will take as input a coarse-scale climate simulation (100km) and will provide as output an accurate one that has consistent large-scale statistics with the reanalysis data. This is not a straightforward task, since due to chaotic divergence there is no direct (i.e. time-wise) correspondence between a coarse-scale climate simulation and a reanalysis dataset. To overcome this problem, we formulate a special protocol for the development of suitable training datasets. The idea is to nudge the coarse-climate model towards the reanalysis data with a very weak penalization term. The result is a coarse-scale simulation that remains close to the reanalysis data-set over large time scales and therefore it is appropriate for training. Using the nudged coarse-scale climate dataset as input and the reanalysis dataset as output we machine learn a correction operator in the form of a LSTM RNN. We subsequently assess the performance of the correction operator on free running coarse-scale climate simulations and show that is indeed able to correct statistical errors which primarily appear in the extreme event regions of the temperature and humidity pdfs. The second part aims to produce small scale features (i.e. high resolution outputs) from coarse-scale inputs. Using reanalysis data for a spatial region of interest we train a machine-learning scheme that naturally ‘splits’ the small-scales into a predictable part, which can be effectively parametrized in terms of the large-scales features, and a stochastic residual, which cannot be uniquely determined using the large-scale information. The later is represented using a conditionally Gaussian process, a choice that allows us to overcome the need for a vast amount of training data, which for climate applications, is naturally limited to a single realization for each spatial location. Using a second round of machine-learning we parametrize, for each location, the covariance of the stochastic component in terms of the large scales. We employ the machine-learned statistics to parsimoniously reconstruct random realizations of the small scales. We demonstrate the approach on reanalysis data involving vorticity over Western Europe and we show that the reconstructed random samples for the small scales result in good agreement to the spatial spectrum, single-point probability density functions, and temporal spectral content. The first part of this work is in collaboration with Alexis-Charalampopoulos (MIT), Dr. Ruby Leung (PNNL) and Dr. Shixuan Zhang (PNNL) and it is supported by the DARPA-ACTM program. The second part is in collaboration with Dr. Zong Yi Wan (MIT), Dr. Boyko Dodov (Verisk Analytics) and Dr. Antoine Blanchard (Verisk Analytics) and it is supported by Verisk Analytics.

Model hierarchies help connect our fundamental understanding of the climate with operational predictions. However, deriving climate model hierarchies using dynamical models of increasing complexity is costly. Additionally, such hierarchies can be challenging to consistently organize since dynamical models may have entirely different outputs that were benchmarked using distinct (and sometimes intractable) data sources. Recently, machine learning approaches have proven invaluable in developing accurate statistical models from data, notably because of their ability to capture nonlinearities and connectivities in time and space ignored by traditional statistical models. Motivated by the possibility of deriving statistical models of varying complexity, from simple analytic equations to complex neural networks, we ask: Can we leverage machine learning to systematically generate a model hierarchy from a single data source? To address this question, we choose four atmospheric science problems for which we have physically-based, analytic models with just a few tunable parameters, and deep learning algorithms whose performance was already established in previous work: cloud cover parameterization, shortwave radiative transfer for numerical weather prediction, scalar flux parameterization in the planetary boundary layer, and subgrid-scale convection parameterization for climate modeling. In each case, we formalize the machine learning-based hierarchy by working in a well-defined, two-dimensional plane: Complexity versus Performance. We choose the number of trainable parameters as a simple definition for complexity, while performance is systematically defined using a single regression metric (e.g., the mean-squared error) calculated for the same outputs over the same dataset that all machine learning models were trained on. During this presentation, we will demonstrate how to use our data-driven hierarchies for two purposes: (1) Data–driven model development; and (2) process understanding. First, each machine learning model of the hierarchy occupies a well-defined (complexity, performance) position as they use the same performance metric. Models that maximize performance for a given complexity unambiguously define a Pareto frontier in (complexity) x (performance) space and can be deemed optimal. Second, optimal models on the Pareto frontier can be compared to reveal which added process/nonlinearity/regime/connectivity/etc. leads to the biggest increase in performance for a given complexity, which facilitates process understanding. For example, using a variational auto-encoder to reconstruct scalar turbulent fluxes in the boundary layer, we can capture non-local vertical transport neglected by traditional eddy-diffusivity schemes. Using a specialized type of convolutional neural network (U-net++) to emulate shortwave radiative heating, we can mostly overcome the biases of a one-stream model of shortwave radiation, notably the negative bias in downward shortwave fluxes at low zenith angles. To show its versatility, we will apply our framework to the data-driven discovery of analytic models, which are interpretable by construction. Combining symbolic regression with sequential feature selection, we derive progressively more complex equations to model cloud area fraction and broadband shortwave radiative fluxes from the local thermodynamic environment. In the case of cloud cover parameterization, a third-order polynomial with only six terms can yield a coefficient of determination as high as 0.65 (compared to 0.9 for the neural network), systematically beating the performance of the widely-used Sundqvist scheme thanks to its ability to capture the effects of cloud condensate mixing ratio on cloud fraction. In summary, our framework can guide the development of Pareto-optimal, data-driven models for weather and climate applications, while furthering process understanding by hierarchically unveiling system complexity.

The atmospheric and oceanic turbulent circulations involve a variety of nonlinearly interacting physical processes spanning a broad range of spatial and temporal scales. To make simulations of these turbulent flows computationally tractable, processes with scales smaller than the typical grid size of weather/climate models have to be parameterized. Recently, there has been substantial interest (and progress) in using deep learning techniques to develop data-driven subgrid-scale (SGS) parameterizations for the climate system. Another approach that is rapidly gaining popularity is to learn the entire spatio-temporal variability of the climate system from data, i.e., developing fully data-driven forecast models or emulators. For either of these approaches to be useful and reliable in practice, a number of major challenges have to be addressed. These include: 1) instabilities or unphysical drifts, 2) learning in the small-data regime, 3) interpretability, and 4) extrapolation to different parameters. Using several setups of 2D turbulence, two-layer quasi-geostrophic turbulence, Rayleigh-Benard convection, and ERA5 reanalysis, we introduce methods to address (1)-(4). The key aspect of some of these methods is combining the spectral analyses of deep neural networks and turbulence/nonlinear physics, as well as leveraging recent advances in theory and applications of deep learning. . In the end, we will discuss scaling up these methods to more complex systems and real-world applications, e.g., for SGS modeling of atmospheric gravity waves and conducting short- and long-term weather forecasting.

The capability of using imperfect statistical reduced-order models to capture crucial statistics in turbulent geophysical systems is investigated. Much simpler and more tractable block-diagonal models are proposed to approximate the complex and high-dimensional turbulent dynamical equations through parametric closure models. New machine learning strategies are proposed to learn the expensive unresolved processes directly from data. A systematic framework of correcting model errors with empirical information theory is introduced, and optimal model parameters under this unbiased information measure can be achieved in a training phase before the prediction. It is demonstrated that crucial principal statistical quantities in the most important large scales can be captured efficiently with accuracy using the reduced-order model in various dynamical regimes of the flow field with distinct statistical structures.