Reasoning language model

Reasoning language models are artificial intelligence systems that combine natural language processing with structured reasoning capabilities. These models are usually constructed by prompting, supervised finetuning (SFT), and reinforcement learning (RL) initialized with pretrained language models.

Prompting

Main article: Prompt engineering

A language model is a generative model of a training dataset of texts. Prompting means constructing a text prompt, such that, conditional on the text prompt, the language model generates a solution to the task. Prompting can be applied to a pretrained model ("base model"), a base model that has undergone SFT, or RL, or both.

Chain of thought

Chain of Thought prompting (CoT) prompts the model to answer a question by first generating a "chain of thought", i.e. steps of reasoning that mimic a train of thought. It was published in 2022 by the Brain team of Google on the PaLM-540B model. In CoT prompting, the prompt is of the form "<Input> Let's think step by step", and the model would respond with a chain of reasoning steps, ended with an answer:$${\text{Input}}\rightarrow \underbrace {{\text{Step}}_{1}\rightarrow {\text{Step}}_{2}\rightarrow \cdots \rightarrow {\text{Step}}_{n}} _{\text{Reasoning chain}}\rightarrow {\text{Answer}}$$Similarly, Tree of Thought prompting generalizes CoT by prompting the model to generate one or more "possible next steps", and then running the model on each of the possible next steps by breadth-first, beam, or some other method of tree search. Similarly, Graph of Thought generalizes CoT so that the reasoning steps form a directed acyclic graph.

Self-consistency decoding performs several chain-of-thought rollouts, then selects the most commonly reached conclusion out of all the rollouts. If the rollouts disagree by a lot, a human can be queried for the correct chain of thought.

Retrieval-augmented generation

Main article: Retrieval-augmented generation

A language model may answer a query by first querying a database of documents using the query. The document retrieval can be via a vector database, summary index, tree index, or keyword table index. Following document retrieval, the LLM generates an output that incorporates information from both the query and the retrieved documents.

Tool use

Language models can perform long reasoning steps by calling external methods, such as numerical recipes, program interpreters, API calls, and so on. This can be prompt-engineered by describing the external methods in-context (an example of in-context learning) or finetuned into the model.

Supervised finetuning

A base model can be finetuned on a dataset of reasoning tasks with example solutions and reasoning traces. The finetuned model would then be able to generate reasoning traces for a given problem.

Reinforcement learning

A pretrained language model can be further trained by RL. In the RL formalism, a generative language model is a policy $\pi$. A prompt specifying a task to solve is an environmental state $x$, and the response of the language model to the prompt is an action $y$. The probability that the language model responds $x$ with $y$ is $\pi (y|x)$.

Training a reasoning language model by RL then consists of constructing a reward model $r(x,y)$ to guide the RL process. Intuitively, a reward model describes how desirable/appropriate/good the response is for the prompt. For reasoning language model, the prompt describes a reasoning task, and the reward would be high if the response solves the task, and low if the response fails to solve the task.

For reasoning language models, the model's response $y$ may be broken down into multiple steps, in which case it is written as $y_{1},y_{2},\dots ,y_{n}$.

Outcome Reward Model

Outcome reward model, or outcome-supervised RM (ORM), is a reward model that computes the reward of a step $r(x,y_{1},\dots ,y_{i})$ determined by the final answer: $r(x,y_{1},\dots ,y_{i})=r(x,y_{n})$. They are also called "verifiers".

For tasks with an answer that is easy to verify, such as word problems in math, the outcome reward can simply be binary: 1 if the final answer is correct, and 0 otherwise. If the answer is not easy to verify programmatically, humans can manually label the answers as correct or not, then the labels can be used to finetune a base model that predicts the human label. For other kinds of tasks, such as creative writing, where verification is difficult, one can train a reward model by finetuning a base model on human preference data, such as used in reinforcement learning from human feedback. A base model can also be finetuned to predict, given a partial thinking trace $x,y_{1},\dots ,y_{m}$, whether the final answer would be correct or not. This can then be used as a binary reward signal.

Process Reward Model

Process reward model, or process-supervised RM (PRM), is a reward model that computes the reward of a step $r(x,y_{1},\dots ,y_{i})$ determined by the steps so far: $(x,y_{1},\dots ,y_{i})$.

Given a partial thinking trace $x,y_{1},\dots ,y_{m}$, a human can be queried as to whether the steps so far are correct, regardless of whether the ultimate answer would be correct. This can then be used as a binary reward signal. As human labels are expensive, a base model can be finetuned to predict the human labels.

Applications

Prompt engineering was discovered in GPT-3 as "few-shot learning", which began a period of research into "eliciting" capacities of pretrained language models. It was then found that a model could be prompted to perform CoT reasoning, which improves its performance on reasoning tasks.

See also

• Generative pre-trained transformer
• Neuro-symbolic AI
• Automated theorem proving
• Automated reasoning
• Large language model

External links

• Fortes, Armando (2025-01-27), atfortes/Awesome-LLM-Reasoning, retrieved 2025-01-27
• Huang, Jie; Chang, Kevin Chen-Chuan (2023-05-26), Towards Reasoning in Large Language Models: A Survey, arXiv, doi:10.48550/arXiv.2212.10403, arXiv:2212.10403, retrieved 2025-01-27