Reasoning language model

Reasoning language models (RLMs) are large language models that are trained further to solve tasks that take several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, and make use of extra computation while answering as another way to scale performance, alongside the number of training examples, parameters, and training compute.

History

2024

In September 2024, OpenAI released o1-preview, an LLM with enhanced reasoning. The full version, o1, followed in December 2024. OpenAI also began sharing results on its successor, o3.

The development of reasoning LLMs has illustrated what Rich Sutton called the "bitter lesson": that scaling compute often outperforms methods that rely on specific human insights. For example, the Generative AI Research Lab (GAIR) explored complex methods such as tree search and reinforcement learning to replicate o1's capabilities. In their "o1 Replication Journey" papers they reported that knowledge distillation (training a smaller model to imitate o1's outputs) worked surprisingly well. This highlighted the effectiveness of distillation in this context.

Alibaba released reasoning versions of its Qwen LLMs in November 2024. In December 2024, the team introduced QvQ-72B-Preview, an experimental visual reasoning model.

In December 2024, Google introduced Deep Research in Gemini, a feature that runs multi-step research tasks.

On December 16, 2024, an experiment with a Llama 3B model showed that by scaling test-time compute, a relatively small model could outperform a much larger Llama 70B model on challenging reasoning tasks. This suggested that better inference strategies can unlock useful reasoning capabilities even in small models.

2025

In January 2025, DeepSeek released R1, a model with comparable performance to o1 at lower cost. The release demonstrated the effectiveness of Group Relative Policy Optimization (GRPO). On January 25, 2025, DeepSeek added a feature to DeepSeek R1 that lets the model search the web while it reasons, making it easier to combine retrieval with reasoning. OpenAI subsequently released o3-mini, followed by Deep Research based on o3. The effectiveness of distillation was shown again by s1-32B, which reached strong performance with budget forcing and scaling methods.

On February 2, 2025, OpenAI released Deep Research, a tool that integrates reasoning and web search in one workflow so users can run complex research that needs several steps and sources. It is based on o3 and can take from 5 to 30 minutes to generate comprehensive reports.

Supervised finetuning

A large language model (LLM) can be fine-tuned on a dataset of reasoning tasks paired with example solutions and step-by-step (reasoning) traces. The fine-tuned model can then produce its own reasoning traces for new problems.

Because human-written traces are costly to collect, researchers have proposed ways to build such datasets automatically. In rejection sampling finetuning (RFT), new reasoning traces are gathered in a loop:

1. Sample a task prompt.
2. Generate many reasoning traces for the prompt.
3. Use a verifier to remove reasoning traces with a wrong final answer, and optionally remove duplicates

Reinforcement learning

A pretrained language model can be further trained with RL. In the RL formalism, a generative language model is a policy $\pi$. A task prompt is an environmental state $x$, and the model's response is an action $y$. The probability that the model responds $x$ with $y$ is $\pi (y|x)$.

Training a reasoning language model with RL means constructing a reward model $r(x,y)$ to guide the RL process. Intuitively, the reward says how good a response is for a prompt. For a reasoning task, the reward is high if the response solves the task and low if it does not.

A response $y$ may be broken-down into multiple steps, written $y_{1},y_{2},\dots ,y_{n}$.

Most recent systems use policy-gradient methods such as Proximal Policy Optimization (PPO) because PPO constrains each policy update with a clipped objective, which stabilises training for very large policies.

Outcome reward model

An outcome reward model, or outcome-supervised RM (ORM), gives the reward for a step $r(x,y_{1},\dots ,y_{i})$ based on the final answer: $r(x,y_{1},\dots ,y_{i})=r(x,y_{n})$. Such models are often called "verifiers".

For tasks with answers that are easy to verify, such as math word problems, the outcome reward can be binary: 1 if the final answer is correct, 0 otherwise. If automatic verification is hard, humans can label answers as correct or not, and those labels can be used to finetune a base model that predicts the human label. For tasks like creative writing, where quality is not simply true or false, one can train a reward model on human ranked preference data, as in reinforcement learning from human feedback. A base model can also be fine-tuned to predict, from a partial thinking trace $x,y_{1},\dots ,y_{m}$, whether the final answer will be correct, and this prediction can serve as a binary reward.

The ORM is usually trained with logistic regression, i.e. by minimizing cross-entropy loss.

Given a PRM, an ORM can be constructed by multiplying the total process reward during the reasoning trace, by taking the minimum, or by other ways of aggregating process rewards. DeepSeek used a simple ORM to train the R1 model.

Process reward model

A process reward model, or process-supervised RM (PRM), gives the reward for a step $r(x,y_{1},\dots ,y_{i})$ based only on the steps so far: $(x,y_{1},\dots ,y_{i})$.

Given a partial thinking trace $x,y_{1},\dots ,y_{m}$, a human can judge whether the steps so far are correct, without looking at the final answer. This yields a binary reward. Because human labels are costly, a base model can be fine-tuned to predict them. The PRM is usually trained with logistic regression on the human labels, i.e. by minimizing the cross-entropy loss between true and predicted labels.

As an example, a 2023 OpenAI paper collected 800K process labels for 75K thinking traces. A labeler saw a trace and marked each step as "positive" if it moved toward a solution, "neutral" if it was not wrong but did not help, and "negative" if it was a mistake. After the first "negative" label, the labeler stopped on that trace and moved to another. The authors argued that labeling up to the first error was enough to train a capable PRM, even though labeling later steps could give richer signals.

To avoid human labels, researchers have proposed methods to create PRM without human labels on the processes. Inspired by Monte Carlo tree search (MCTS), the Math-Shepherd method samples multiple continuations until the end, starting at each reasoning step $y_{i}$, and set the reward at that step to be either ${\frac {\#{\text{(correct answers)}}}{\#{\text{(total answers)}}}}$ in the case of "soft estimation", or ${\begin{cases}1&{\text{if one of the answers is correct}}\\0&{\text{else}}\end{cases}}$ in the case of "hard estimation". This creates process rewards from an ORM, which is often easier or cheaper to construct. A PRM can then be trained on these labels. Some work has tried a fully MCTS approach.

One can also use an ORM to implicitly construct a PRM, similar to direct preference optimization.

Guided sampling

A trained ORM can be used to pick the best response. The policy generates several responses, and the ORM selects the best one. This implements a simple form of test-time compute scaling ("best-of-N").

A trained PRM can guide reasoning by a greedy tree search: the policy proposes several next steps, the PRM picks one, and the process repeats. This mirrors using an ORM to pick a whole response. Beam search performs better than greedy search.

Lookahead search is another tree search method. The policy proposes several next steps, then makes a short rollout for each. If a solution is found during rollout, the search stops early. Otherwise, the PRM scores each rollout, and the step with the highest score is chosen.

Self-consistency can be combined with an ORM. The model generates multiple answers, and the answers are clustered so that each cluster has the same final answer. The ORM scores each answer, scores in each cluster are summed, and the answer from the highest-scoring cluster is returned.

Benchmarks

Reasoning models generally score higher than non-reasoning models on many benchmarks, especially on tasks requiring multi-step reasoning.

Some benchmarks exclude reasoning models because their responses take longer and cost more.

Humanity's Last Exam

The HLE benchmark tests expert-level reasoning across mathematics, humanities, and the natural sciences, and shows large performance gaps between models. State-of-the-art reasoning models score low on HLE, leaving room to improve. For example, the full reasoning model o3 reached 26.6%, while the lighter o3-mini-high (on text-only questions) reached 13%.

AIME

On the American Invitational Mathematics Examination (AIME), a difficult math competition, non-reasoning models usually solve under 30% of problems. Models that use reasoning methods score between 50% and 80%. While OpenAI's o1 maintained or slightly improved its accuracy from reported 2024 results to 2025 AIME results, o3-mini (high) reached a higher accuracy (80%) at a much lower cost (about 12 times cheaper).

o3-mini performance

According to OpenAI's January 2025 report on o3-mini, adjusting "reasoning effort" significantly affects performance, especially for STEM tasks. Moving from low to high reasoning effort raises accuracy on AIME 2024, GPQA Diamond, and Codeforces, typically by 10–30%. With high effort, o3-mini (high) achieved 87.3% on AIME (different from the MathArena AIME benchmark), 79.7% on GPQA Diamond, 2130 Elo on Codeforces, and 49.3 on SWE-bench Verified.

Drawbacks

Computational cost

Reasoning models often need far more compute while answering than non-reasoning models. On AIME, they were 10 to 74 times more expensive than non-reasoning counterparts.

Generation time

Reasoning increases response time, with current models taking from a few seconds to several minutes to answer. As depth of reasoning grows, future models may need even longer.

Models

OpenAI

• o4-mini
• o3 and o3-mini
• o1 and o1-preview

Gemini

• 2.5 Pro and Flash
• 2.0 Flash Thinking

DeepSeek

• R1 (based on V3)
• R1-Lite-Preview (test version based on V2.5)

Qwen

• QvQ-72B-Preview — an experimental visual reasoning model launched on December 24, 2024, which integrates image understanding with verbal chain-of-thought reasoning.
• QwQ-32B-Preview — an experimental text-based reasoning model released in late November 2024 that emphasizes complex, step-by-step analysis.

Anthropic

• Claude Sonnet 3.7 has an adjustable amount of 'thinking' tokens.

Mistral AI

• Magistral (medium & small)

xAI

• Grok 3
• Grok 4

Hugging Face

• OlympicCoder-7B & 32B, as part of reproducing the R1 training openly (Open R1 project).

See also

• Automated reasoning
• Reflection (artificial intelligence)
• Large language model

External links

• Fortes, Armando (2025-01-27), atfortes/Awesome-LLM-Reasoning, retrieved 2025-01-27
• Huang, Jie; Chang, Kevin Chen-Chuan (2023-05-26), Towards Reasoning in Large Language Models: A Survey, arXiv:2212.10403
• Besta, Maciej; Barth, Julia; Schreiber, Eric; Kubicek, Ales; Catarino, Afonso; Gerstenberger, Robert; Nyczyk, Piotr; Iff, Patrick; Li, Yueling (2025-01-23), Reasoning Language Models: A Blueprint, arXiv:2501.11223