Bayesian Statistics and Statistical Learning

We consider two types of rate-distortion problems associated with Bayesian learning. Formulating a rate-distortion problem by the distortion measure defined by the pointwise regret of the model parameter and another rate-distortion problem by replacing the empirical expectation with the true expectation, we show the relationships between the asymptotic behaviors of the rate-distortion functions and the learning coefficients of Bayesian learning. Furthermore, the two rate-distortion bounds for a fixed prior derive a relationship between the expected regret and Gibbs generalization error.

One of the key shifts in statistical machine learning over the past decade has been the transition from handcrafted features to automated, data-driven representation learning, typically via deep neural networks. As these methods are being used in high stakes settings such as medicine, health care, law, and finance where accountability and transparency are not just desirable but often legally required, it has become necessary to place representation learning on a rigourous scientific footing. In this talk we will re-visit the statistical foundations of representation learning from the lens of nonparametric latent variable models, and discuss how even basic statistical properties such as identifiability and consistency are surprisingly subtle. We will also discuss new results characterizing the optimal sample complexity for learning simple nonparametric mixtures, which turns out to have a nonstandard super-polynomial bound. Time permitting, we will end with applications to deep generative models that are widely used in practice.

This talk is based on joint work with Wai Ming Tai and Ruiyi Yang.

Artificial neural networks are regarded more and more as universal learning machines, but in some cases, such as when learning the interaction between physical particles it is important to explicitly code into them known invariances, such as invariance to translation, rotation, etc.. The field of group equivariant neural networks has developed to address this challenge. In this talk I will give an overview of this quickly growing field, focusing on its foundations in representation theory, and giving some general prescriptions for building neural architectures that are equivariant to the action of any given compact group. I will demonstrate the methodology with some applications in image recognition, graph learning and force field learning for molecular dynamics.

Privacy preserving mechanisms such as differential privacy inject additional randomness in the data, beyond the sampling mechanism. Ignoring this additional noise can lead to inaccurate and invalid inferences. In this talk, we present two examples to perform statistical inference using data released from a differentially private mechanism. The first example centers around the beta model of random graphs, where the degree sequence is a sufficient statistic. Here, we release a noisy sufficient statistic, and then “de-noise” it to perform maximum likelihood estimation. The second example centers around a naive bayes classifier. Here, we explicitly incorporate the privacy mechanism into the likelihood function by treating the original data as missing. However, the corresponding likelihoods are intractable and we derive fast and accurate variational approximations to tackle such intractable likelihoods that arise due to privacy. 

We study the learning of finite mixture modeling in a singular setting,
where the number of model parameters is unknown. Simply overfitting
without suitable regularization of the model can result in highly inefficient
parameter estimation behavior.

To achieve the optimal rate of parameter estimation, we propose a
general framework for estimating the mixing measure arising in
finite mixture models, which we term minimum $Phi$-distance estimators.  
We establish a general theory for the general minimum $Phi$-distance
estimator, which involves obtaining sharp probability bounds on the
estimation error for the mixing measure in terms of the suprema of
the associated empirical processes for a suitably chosen function class
$Phi$. The theory makes clear the dual roles played by the function
class $Phi$, rich enough to induce necessary strong identifiability
conditions and small enough to produce optimal rates for parameter
estimation.

Our framework not only includes many existing estimation methods as special
cases but also results in new ones. For instance, it includes the minimum
Kolmogorov-Smirnov distance estimator as a special case, but also extends
to the multivariate setting.  It includes the method of moments as well,
while extending existing results for Gaussian mixtures to a larger family
of probability kernels. Moreover, the minimum $Phi$-distance estimation
framework leads to new methods applicable to complex (e.g., non-Euclidean)
observation domains.  In particular, we study a novel minimum distance
estimator based on the maximum mean discrepancy (MMD), a particular
$Phi$-distance that arises in a different context (of learning
with reproducing kernel Hilbert spaces).  

This work is joint with Yun Wei and Sayan Mukherjee.

Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients and explore the role of depth. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK. The talk is based on joint work with Michael Murray, Hui Jin, and Benjamin Bowman. 

In many machine learning tasks, the goal is to learn an unknown function which has some known group symmetries. Equivariant machine learning algorithms exploit this by devising architectures (=function spaces) which have these symmetries by construction. Examples include convolutional neural networks which respect translation symmetries, neural networks for graphs or sets which respect their permutation symmetries, or neural networks for 3D point sets which additionally respect Euclidean symmetries.

At the very origin of Algebraic Statistics was the 1998 theorem by
Diaconis and Sturmfels showing the existence of Markov bases. They introduced a new sampling algorithm for a broad class of problems, showing how to explicitly construct a Markov chain over the support of the conditional distribution given the log-linear exponential family’s sufficient statistic. The chain is built using combinatorial moves that have analogues in polynomial algebra and polyhedral geometry. Each instance inherits a finite set of moves necessary to obtain an irreducible Markov chain for any value of the sufficient statistic. This has been applied for example in the context of contingency tables. After 25 years of this influencial work this talk discusses what we do not know about the complexity and structure of Markov bases and new applications in machine learning.

All new results are joint work with Felix Almendra Hernandez and Sonja Petrovic and with Yue Wu.

We use machine learning techniques to develop tests to separate types of quantum entanglement. In particular, I will describe an artificial neural network model we used to learn membership on certain algebraic varieties. Our results show that it is possible to learn membership on varieties (like a hyperdeterminantal hypersurface) where traditional interpolation methods would be infeasible. 

This is joint work with Hamza Jaffali who works at ColibrITD, a quantum information startup in Paris, France.

The Gauss–Manin connection of a family of hypersurfaces governs the change of the period matrix along the family. This connection can be complicated even when the equations defining the family look simple. When this is the case, it is expensive to compute the period matrices of varieties in the family via homotopy continuation. We train neural networks that can quickly and reliably guess the complexity of the Gauss-Manin connection of pencils of hypersurfaces. As an application, we compute the periods of 96% of smooth quartic surfaces in projective 3-space whose defining equation is a sum of five monomials; from the periods of these quartic surfaces, we extract their Picard lattices and the endomorphism fields of their transcendental lattices.

In this talk I will discuss covering numbers of real algebraic varieties and applications to data science.  Specifically, we upper bound the number of balls of radius epsilon needed to cover a real variety, image of a polynomial map or semialgebraic set in Euclidean space, in terms of the degrees of the relevant polynomials and number of variables.  The bound remarkably improves the best known general bound, and its proof is much more straightforward.  On the application side, we control covering numbers of low rank CP tensors, bound the sketching dimension for polynomial optimization problems, and bound the generalization error for deep rational and ReLU neural networks.  Joint work with Yifan Zhang (UT Austin), see arXiv:2311.05116.