Advances in Machine Learning and Spatial and Spatio-Temporal Statistics

Machine learning methods have enjoyed a wide popularity in the last few years. While in most cases machine learning methods have been praised for their ability to capture non-linear effects of different covariates on the mean structure of the data, in recent years interest has been placed in extending these approaches to also account for spatial dependence. In this talk we investigate whether the predictions generated by these methods are still accurate when the data are a realization of a non-stationary spatial process with non-stationarity not only in the mean function but also in the second-order structure. Through simulation experiments we highlight situations in which these methods seem to perform less well, probably due to over-smoothing. We propose a solution that addresses the spatial over-smoothing and we illustrate our approach with an application in Earth System sciences. 

Spatially-varying models capture relationships between explanatory variables and outcomes that change across a spatial domain, making them essential for addressing spatial problems. In contrast, traditional machine and deep learning models rely on a single, global fit, often failing to account for spatial heterogeneity. To bridge this gap, we introduce a spatially-weighted decorrelation transformation, inspired by local and geographically weighted regression principles. This approach enables machine and deep learning models to be applied locally at each spatial location while maintaining a smooth, continuous transition across the domain. The result is a spatially-adaptive machine learning framework that enhances predictive performance in the presence of spatial non-stationarity. 

In this talk, I will discuss the contributions and ongoing research of my Environmental Statistics Research Group in the area of spatio-temporal statistics, with a particular focus on leveraging deep learning and high performance computing for spatio-temporal analysis in Geo-Environmental Data Science. I will introduce the developed innovative software tools such as ExaGeoStat, ParallelVecchiaGP, and DeepKriging, which support the analysis of large-scale geostatistical datasets. During this presentation, I will also showcase environmental applications to air quality modeling and prediction. 

Nonparametric machine learning (ML) models have long been used as surrogates to replicate the turbulence simulations of more computationally expensive and memory demanding direct numerical simulations (DNS) methods (Jordan and Mitchell 2015, Brenner, Eldredge and Freund 2019). In the ML literature, convolutional neural networks (CNN), which focus on capturing spatial relationships, and recurrent neural networks (RNN), instead best suited for temporal data, have been used to model the spatiotemporal processes underlying turbulent flows (LeCun, Bengio and Hinton 2015, Tealab 2018). Since neural network (NN) parameters are, traditionally, learned by minimizing a loss function which is agnostic of the physics surrounding the problem, recent studies have presented NNs that can account for the physical laws governing a given system of interest (Greenwood 1991). These physics-informed neural networks (PINNs) incorporate this knowledge in the form of an additional constraint on the objective function, akin to ridge regression, so that model predictions are physically consistent (Hoerl and Kennard 2000, Raissi, Perdikaris and Karniadakis 2019). PINNs have indeed been shown to perform better than NNs in many applications where data is difficult to collect but information on the process in the form of a PDE is available (Zhang and Zhao 2021). PINNs have also been successfully employed as mesh-free PDE solvers for highly idealized systems such as the Burgers' equation, describing the behavior of a one-dimensional fluid following a shock, laminar flows, i.e., non-turbulent, and the Reynolds-averaged Navier Stokes (RANS) equations, which consider the mean characteristics (in time) of a turbulent flow (Raissi, Perdikaris and Karniadakis 2019, Rao, Sun and Liu 2020).

Increased attention has been given to convolutional recurrent neural networks (CRNNs) in spatiotemporal studies on turbulent flows, as they are often employed to address RNNs' main drawbacks: they require large amounts of data, as is almost always the case in turbulence data sets, and significant computational resources for adequate parameter estimates (Bianchi, et al. 2017). CRNNs address both issues by modeling the temporal structure of physical processes through a latent space representation (of lower dimensions) of the original data. This approach has yielded successful results in two-dimensional flow past a cylinder as well as low levels of turbulence (Akbari, Akbari and San 2022, Clark Di Leoni, et al. 2023). CRNNs are in fact composed of three parts, a convolutional encoder, which extracts the spatial features of the original data and projects it onto a latent space, the RNN, which models the temporal development of the flow in latent space, and a convolutional decoder, which projects the latent space representation of the output back onto the original dimensions of the data. Initially conceived for image classification and denoising tasks, the convolutional elements are often referred to as a convolutional autoencoder (CAE), which, when applied to large spatiotemporal data sets, is often used as a reduced order modeling (ROM) technique (Y. Zhang 2018).

In this work we combine the aforementioned ML approaches into a physics-informed convolutional recurrent neural network (PI-CRNN) to model long sequences of the Rayleigh-Bénard convection (RBC) spatiotemporal process, a type of turbulent flow where warm (i.e., less dense) particles move from a flat surface at the bottom of the spatial domain to a cold one, at the top, and vice-versa for cold (i.e., more dense) particles. The PDEs governing this system include mass conservation, momentum conservation, and energy conservation. Depending on the physical parameters of choice, e.g., fluid viscosity and density, RBC may manifest itself as a slowly-developing periodic convection from the bottom to the top, i.e., more predictable, or as a highly chaotic turbulent flow, which is the focus of this work. The convolutional component captures the spatial relationships, the recurrent component models the temporal evolution of long sequences (in latent) of RBC, and the physics-informed component ensures that inference is constrained by the governing PDEs. The goal of this work is to present a statistical model able to forecast long, physically-consistent sequences of RBC at a fraction of the computational and memory demand of DNS, as the temporal framework that we employ is similar to that of language translation (Cho, et al. 2014).
 

We consider a high dimensional and high resolution time series dataset of human gait for the purpose of predicting and preventing falls. This data was collected via motion capture of 13 individuals walking across a variety of artificial terrains. The angle of 9 joints were continuously measured through a total of 15000 strides. Our goal is to develop an approach for modeling the individual distribution of gait patterns, so that deviations from that individual distribution can serve as warning signs of increased fall risk.

We propose a Langevin Diffusion Model with a nonparametric stationary distribution, which we model using an Infinite Dimensional Exponential Family. This model can flexibly estimate individual-level stationary distributions of gait style and also permits detection of departures from an individual's movement profile. We use this model to simulate synthetic gait data, and also to develop methods for identifying In turn, this prediction apparatus will enable simulation of gait data for engineering applications and potentially improve health outcomes through fall prevention.