A lot of effort has gone into improving the capabilities of large language models (LLMs) in recent years. We may now be close to exhausting what can be achieved with brute-force methods like increasing the size of training datasets and upping the number of parameters in a model. When an LLM has already been trained on the text of the entire internet, there is not much more digital information that can be added. And with models already surpassing a trillion parameters, it is growing increasingly impractical from the perspective of energy consumption and available computational resources to make them any larger.
Test-time scaling is an interesting new approach that may keep the ball moving forward. It enhances a model’s performance by increasing compute time during inference rather than solely relying on extensive pretraining. This concept has been gaining a lot of traction since OpenAI’s o1 model demonstrated strong reasoning performance through test-time scaling techniques. However, OpenAI’s interpretation of “open” diverges from common understanding, so the methodology was not made public.
Test-time scaling increases model accuracy (📷: N. Muennighoff et al.)
This led a team of researchers at Stanford University to take a crack at developing their own test-time scaling solution with strong reasoning performance. Their method, called budget forcing, allows them to control how much computational effort an LLM expends during inference, essentially managing the length and depth of its reasoning process. The method involves either forcing a model to stop reasoning early, or encouraging it to think longer when it would otherwise try to conclude its answer. This approach has shown promising results in getting models to double-check their reasoning and correct errors that might otherwise go unnoticed.
To test the effectiveness of budget forcing, the researchers created a small but carefully curated dataset called s1K, consisting of 1,000 questions paired with detailed reasoning traces. These questions were selected based on three key factors — difficulty, diversity, and quality — ensuring that the model learns from a well-balanced dataset. The model used for testing, s1-32B, was trained using supervised fine-tuning on this dataset and then evaluated with budget forcing applied during inference.
The results were quite impressive. The s1-32B model, equipped with budget forcing, outperformed OpenAI’s o1-preview model on competitive math benchmarks, including MATH and AIME24, by up to 27%. This demonstrates that test-time scaling, when properly controlled, can significantly enhance a model’s reasoning ability without requiring an increase in training data or model size.
The s1K dataset is efficient, training accurate models on few samples (📷: N. Muennighoff et al.)
The team also compared their method to alternative test-time scaling techniques such as conditional length control and rejection sampling. In the process, they introduced three metrics for measuring effectiveness: controllability (how well the method regulates computational effort), scaling efficiency (how performance improves with increased compute), and overall performance. Budget forcing performed better across all three criteria, confirming its effectiveness in enhancing LLM reasoning capabilities.
Moving forward, this approach could play a role in making AI models smarter, more reliable, and more efficient. Toward that goal, the research findings, along with the dataset and code, have been made open-source to allow others in the AI community to build on the work.
