Two Frontiers in AI Scaling

Music City Center 

Large but finite internet data, which has powered AI scaling in the past decade, is rapidly becoming depleted, motivating people to search for new frontiers such as test-time scaling. This talk will introduce two recent projects that probe this new frontier. First, we will present synthetic continued pretraining, a technique that converts excess compute into a statistical signal, which scales the model towards better data-efficiency. Next, we will introduce the s1-32B model, an open-source test-time scaling mechanism which uses minimal resources and evidences a transparent understanding of scaling. Overall, we will see that there are exciting opportunities around (continued) pretraining data scaling, and a cautiously optimistic path toward test-time scaling.

Synthetic data

Language model