Colloquium Series: Vidya Muthukumar

Our upcoming event for the Statistics Colloquium Series is scheduled for Monday, April 6 from 12:00 – 1:00pm (ET) and will be an in-person presentation at Maxwell-Dworkin 134A/B. Lunch will be provided to guests following the talk. This week's speaker will be Vidya Muthukumar of Georgia Institute of Technology's School of Electrical and Computer Engineering and the School of Industrial and Systems Engineering.

Two vignettes on kernel interpolation and approximation, and applications to feature learning

Abstract: It is increasingly believed that a key component of the unique generalization abilities of overparameterized neural networks involves adapting to nonlinear, low-dimensional structure in data (commonly called “feature learning”). A compelling interpretation of this phenomenon would arise by viewing the neural network as a data-dependent kernel, where the feature map is itself learned as a function of the training data and adequately captures the hidden lower-dimensional structure. Formalizing this interpretation is challenging due to outstanding challenges in the error analysis of kernel methods, especially when the training data is perfectly fit and/or the data is high-dimensional.

In this talk, we present two perspectives on analyzing kernel ridge/less regression in the modern high-dimensional regime. The first assumes explicit access to the eigenfunctions and eigenvalues of the kernel integral operator and derives upper bounds on the test regression and classification error purely as a function of the eigenvalues under a mild “bounded orthonormal system” assumption on the eigenfunctions. Under stronger assumptions on the eigenfunctions, these bounds match the error rates of an equivalent overparameterized linear model with whose covariates are Gaussian/independent sub-Gaussian. The second vignette considers dot-product kernel matrices on general, anisotropic high-dimensional data, where an eigendecomposition is typically not available except in special cases. We consider the sample size and data dimension to be polynomially related and provide a new approach to approximating the empirical kernel matrix by combining de la Pena’s decoupling inequality and the non-commutative Khintchine inequality. This approach recovers previous optimal approximation results for Boolean/spherical data (via a simplified proof) and results in a tighter lower bound on the bias of dot-product kernel regression on anisotropic Gaussian data. We conclude the talk with a preliminary application of these results to characterize the noise overfitting error of neural networks in the feature learning regime.

Speaker's Bio:

 Vidya Muthukumar is the Harold R. and Mary Anne Nash Early Career Professor and Assistant Professor in the School of Electrical and Computer Engineering and H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. Her broad interests are in game theory, online and statistical learning. She is particularly interested in designing learning algorithms that provably adapt in strategic environments, fundamental properties of overparameterized ML models, and the foundations of multi-agent decision-making. Before joining Georgia Tech, she spent a semester at the Simons Institute for the Theory of Computing as a research fellow for the program “Theory of Reinforcement Learning.” She is the recipient of an Amazon Research Award, NSF CAREER Award, Adobe Data Science Research Award and Simons-Berkeley Research Fellowship.