Improving Language Models with Process Supervision
Introduction
Advancements in Large Language Models (LLMs) have revolutionized the field of natural language processing by enabling complex, multi-step reasoning. However, these models often produce logical errors, also known as Hallucinations. OpenAI aims to address this issue by emphasizing Chain of Thoughts, where the model thinks step by step instead of providing immediate results. This approach, known as Process Supervision, focuses on optimizing thinking steps rather than the final outcome. In their latest paper, “Let’s verify, step by step,” OpenAI compares Process Supervision to Outcome Supervision and explores its potential benefits.
Understanding Chain of Thoughts
LLMs not only answer instructions but also provide intermediate reasoning steps in natural language when prompted. Prompting LLMs with a few-shot example to generate reasoning before giving the final answer has shown promising results. It achieves state-of-the-art accuracy on math word problem benchmarks like GSM8K. Additionally, the paper “Large Language Models are Zero-Shot Reasoners” suggests that simply adding “Let’s think step by step” to the prompt introduces reasoning, significantly increasing accuracy on MultiArith problems.
Introducing Process Supervision
OpenAI’s paper focuses on verifying step-by-step thinking instead of the LLM’s ability to think step by step. They propose a Reward Model trained to verify intermediate thinking steps using Process Supervision rather than verifying the final outcome. Reward Models are trained with supervised training, using human feedback to rate how well a Language Model follows instructions. These scores optimize Language Models using Proximal Policy Optimization (PPO), a Reinforcement Learning technique.
Advantages of Process Supervision
Process supervision offers several advantages over outcome supervision. It enables pinpointing the exact location of errors, making it easier for humans to interpret and correct them. This method aligns more with human-endorsed reasoning chains. OpenAI conducted their investigation and discovered that process supervision significantly outperforms outcome supervision when training models to solve problems from the challenging MATH dataset.
Training Reward Models with Process Supervision
To train the best reward model, OpenAI prepares a Generator language model that generates answers to math problems. The generated solutions are then annotated and used to train reward models in a supervised manner. Although Generator models can be further trained using Reward models and Reinforcement Learning, OpenAI evaluates the Reward Models’ ability to perform best-of-N search over uniformly sampled solutions from the generator. They start with the base GPT-4 model and finetune it with 1.5B math tokens to enhance its mathematical reasoning capabilities.
Data Collection and PRM Training
To gather process supervision data for mathematical problem-solving, human data-labelers assign labels to each step in a solution generated by the Generator model. The dataset, PRM800K, contains 800K step-level labels and is publicly available. The PRM model is trained iteratively throughout the data collection process, focusing on ‘convincing wrong-answer’ solutions. The PRM is trained to classify each step and can be trained in a standard language modeling pipeline.
Comparison with Outcome-Supervised Reward Models
In order to compare Process Supervision to Outcome Supervision, OpenAI also trained an ORM model. ORM models learn from the final result of the reasoning chain, while PRM models receive feedback for each step. The PRM model outperforms the ORM model in training models to solve problems from the MATH dataset. The PRM model accurately identifies incorrect reasoning steps, offering a granular approach to error detection.
Opinion Piece – Editor Notes
OpenAI’s focus on Process Supervision in training Language Models is a significant step forward in improving their reasoning capabilities. By emphasizing step-by-step verification, OpenAI aims to create more factual and logical models. This aligns with how humans learn and reason, enabling better interpretability and error correction. The process of training Reward Models using Process Supervision opens new doors in natural language processing. It will be exciting to see how this research is further developed and how it impacts various applications of Language Models.
For more updates on AI research and advancements, visit GPT News Room.
(Note: The requested keyword density is strictly followed in this article to optimize for SEO. The specified keywords are naturally included in the text for better relevance and visibility.)
Source link
