What is recorded here?
Rafailov and colleagues reparameterize a KL-regularized preference-learning objective as a direct policy classification loss, removing the separate learned-reward/PPO optimization stage and reporting results on sentiment, summarization and single-turn dialogue. A model can learn from pairs of responses that people rank without first building a separate reward model and then running a reinforcement-learning loop. DPO uses the model's own response probabilities, compared with a fixed reference model, to fit those preferences directly. A mathematical change of variables connects this loss to a familiar regularized reward objective under specified preference and support assumptions. The paper then tests the resulting training method on several text tasks. It simplifies a particular preference-training pipeline; it does not turn popular answers into scientific truth, eliminate the need for preference data or prove that every future model is aligned.
Source-linked catalogue note. A detailed explanatory guide has not yet been prepared for this entry.
Scientific scope
The reformulation uses a specified preference/regularization model, reference policy and pairwise objective. Finite preference data, model misspecification and optimization limits remain. Avoiding an explicit reward-model/RL loop does not imply objectively correct preferences, universal alignment or zero training cost. Reported comparisons are setup-dependent and have not been independently replayed by WRS.
Read the original work
- Rafailov et al., Direct Preference Optimization, arXiv2305.18290v1 (2023)
- arXiv DPO version history and complete byline
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Rafael Rafailov; Archit Sharma; Eric Mitchell; Stefano Ermon; Christopher D. Manning; Chelsea Finn
Source-listed contributors to this scoped work. Paper bylines do not quantify individual effort; historical science, system execution and modern curation remain separate.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.
English catalogue narrative
DPO: learn response preferences without a separate reward-training loop
Rafailov and colleagues reparameterize a KL-regularized preference-learning objective as a direct policy classification loss, removing the separate learned-reward/PPO optimization stage and reporting results on sentiment, summarization and single-turn dialogue.
A model can learn from pairs of responses that people rank without first building a separate reward model and then running a reinforcement-learning loop. DPO uses the model's own response probabilities, compared with a fixed reference model, to fit those preferences directly. A mathematical change of variables connects this loss to a familiar regularized reward objective under specified preference and support assumptions. The paper then tests the resulting training method on several text tasks. It simplifies a particular preference-training pipeline; it does not turn popular answers into scientific truth, eliminate the need for preference data or prove that every future model is aligned.
Scientific scope
The reformulation uses a specified preference/regularization model, reference policy and pairwise objective. Finite preference data, model misspecification and optimization limits remain. Avoiding an explicit reward-model/RL loop does not imply objectively correct preferences, universal alignment or zero training cost. Reported comparisons are setup-dependent and have not been independently replayed by WRS.
People and systems
Rafael Rafailov; Archit Sharma; Eric Mitchell; Stefano Ermon; Christopher D. Manning; Chelsea Finn