The Probabilitas Seminar series focuses on high-dimensional problems that combine statistics, probability, information theory, computer science, and other related fields. The upcoming seminar takes place on Friday, November 15, from 10:30-11:30am EST. The talk will be hybrid, both in-person in Science Center 316 and on Zoom (please contact the department for Zoom information). This week's guest will be Qi Lei of NYU.
Title: Efficient and Distribution-aware Model and Data Pruning
Abstract: In the foundation model paradigm, pruning data and model sizes without compromising performance is essential for efficiency, energy, and memory savings. This talk presents two approaches to data and model pruning, each using distribution-aware metrics for efficient, single-shot selection of data samples or model components. In the first part, I introduce Sketchy Moment Matching (SkMM), a scalable data selection method for finetuning that balances variance and bias by matching the second moment of the original data distribution within a sketched low-dimensional space. SkMM enables fast, theoretically sound data selection while retaining critical distributional properties. In the second part, I discuss a structured pruning method for large language models that preserves model outputs. This method leverages a depth-2 pruning structure and input-distribution-aware metrics to achieve efficient model pruning without retraining. Together, these methods illustrate how distribution-aware metrics and greedy selection can yield effective pruning strategies that preserve model performance and reduce computational costs across various machine learning tasks.
LOCATION:Science Center 316 STATUS:CONFIRMED DTSTART:20241115T153000Z DTEND:20241115T163000Z END:VEVENT END:VCALENDAR