Probabilitas Seminar: Atish Agarwal
Date and Time
Location
The Probabilitas Seminar series focuses on high-dimensional problems that combine statistics, probability, artificial intelligence, information theory, computer science, and other related fields. The upcoming seminar takes place on Tuesday, October 14, at 1:30pm in Science Center 705. This week's speaker will be Atish Agarwal, research scientist at Google Deep Mind.
Universal behavior in compute-optimal learning curves
Much recent progress in machine learning has been driven by practical scaling recipes, which predict how to pick things like model size, dataset size, and hyperparameters to reach the lowest test/eval loss for a fixed compute budget. In this talk we explore how these practical considerations give rise to theoretically interesting phenomena. We present the discovery of "Supercollapse", which reveals universal shapes of compute-optimally trained networks at different scales. We demonstrate empirically that a simple rescaling of the learning curves causes collapse onto a universal form with remarkable small deviations, and that this collapse relies heavily on the power-law nature of the compute-optimal Pareto frontier. We then show that a relatively simple model can capture aspects of the collapse quantitatively. This work suggests that practical procedures in machine learning are a rich source of theoretically interesting and tractable problems.
Join Zoom meeting
https://harvard.zoom.us/j/98755337221?pwd=LXgCEWsrWqmapYi0khpQu5XqamdkWq.1
Password: 388485
Join by telephone (use any number to dial in)
+1 929 436 2866
+1 301 715 8592
+1 305 224 1968
+1 309 205 3325
+1 312 626 6799
+1 646 931 3860
+1 689 278 1000
+1 719 359 4580
+1 253 205 0468
+1 253 215 8782
+1 346 248 7799
+1 360 209 5623
+1 386 347 5053
+1 507 473 4847
+1 564 217 2000
+1 669 444 9171
+1 669 900 6833
International numbers available: https://harvard.zoom.us/u/ap49XCxyW
One tap mobile: +19294362866,,98755337221# US (New York)
Join by SIP conference room system
Meeting ID: 987 5533 7221