Date:
Location:
The Probabilitas Seminar series focuses on high-dimensional problems that combine statistics, probability, information theory, computer science, and other related fields. The upcoming seminar takes place on Friday, February 23, from 10:30-11:30am EST. This week's guest will be Murat Erdogdu of the Univesity of Toronto.
Title: Feature Learning in Two-layer Neural Networks: The Effect of Data Covariance
Abstract: We study the effect of gradient-based optimization on
feature learning in two-layer neural networks. We consider a setting
where the number of samples is of the same order as the input
dimension and show that, when the input data is isotropic, gradient
descent always improves upon the initial random features model in
terms of prediction risk, for a certain class of targets. Further
leveraging the practical observation that data often contains
additional structure, i.e., the input covariance has non-trivial
alignment with the target, we prove that the class of learnable
targets can be significantly extended, demonstrating a clear
separation between kernel methods and two-layer neural networks in
this regime.