Our upcoming event for the Statistics Department Colloquium Series is scheduled for Monday, March 24 from 12:00 – 1:00pm (ET) and will be an in-person presentation Science Center 300H. Lunch will be provided to guests following the talk. This week's speaker will be Yury Polyanskiy of MIT's Electrical Engineering and Computer Science.
Optimal Quantization for LLMs and Matrix Multiplication
The main building block of large language models is matrix multiplication, which is often bottlenecked by the speed of loading these matrices from memory. A number of recent quantization algorithms (SmoothQuant, GPTQ, QuIP, SpinQuant etc) address this issue by storing matrices in lower precision. We derive optimal asymptotic information-theoretic tradeoff between accuracy of the matrix product and compression rate (number of bits per matrix entry). We also show that a non-asymptotic version of our construction (based on nested Gosset lattices and Conway-Sloan decoding), which we call NestQuant, reduces perplexity deterioration almost three-fold compared to the state-of-the-art SpinQuant (on 4-bit quantized Llama-3-8B).
Based on a joint work with Or Ordentlich (HUJI), Eitan Porat and Semyon Savkin (MIT EECS).
LOCATION:Science Center 316 STATUS:CONFIRMED DTSTART:20250324T160000Z DTEND:20250324T170000Z END:VEVENT END:VCALENDAR