Average examples instead of trusting one pair
Contrastive Activation Addition builds steering directions from many paired examples of a behavior. It averages differences in internal activations and applies the result while a model generates text. The original tests Llama 2 models. It is closely related to earlier prompt-pair Activation Addition, with a distinct dataset-based construction and evaluation.
A small average shows the idea
For an invented arithmetic illustration, suppose three example differences are (2, 0), (4, 2) and (3, 1). Their average is (3, 1). Combining examples can reduce some example-specific variation. It cannot eliminate a bias shared by all three examples, and averaging alone does not prove that a direction captures the intended behavior.
The comparison is about the increment
The paper contrasts its many-example construction with single-pair steering and other intervention methods. Source-reported benefits concern specified behavioral evaluations. The existing ideas of internal activations, linear directions and inference-time steering are not counted again as entirely new discoveries. The useful question is what the new construction makes more reliable or easier to study. For an invented averaging example, suppose two prompt pairs give differences (2, 0) and (4, 2). Their mean is (3, 1). Averaging can reduce variation specific to one example, but cannot remove a bias shared by both pairs. A separate evaluation set matters because reusing the construction examples would make the intervention look more general than the check establishes.
An evaluator is part of the experiment
Some behavior measurements use another model as a judge. Its rating is a measurement under a particular rubric, not independent proof of a human psychological trait. Effects can depend on the examples, the model, intervention strength and the task. Side effects and transfer to new prompts therefore need their own evidence.
What to check next
Trace one reported behavioral comparison to the examples used to form the direction and to the evaluation procedure. Keep construction data and test data separate in your notes. Explain which result was measured and which broader conclusion would need a new experiment. This is a useful source-checking task without claiming that you reproduced the model run.
Scientific scope
This uses contrastive examples, activation access and selected intervention parameters. Behavioral changes and side effects depend on the model, task and steering strength; pair-averaged construction is distinct from a single-prompt activation difference.
Read the original work
Sources and reading locations for this explanation
- Steering Llama 2 via Contrastive Activation Addition
Pinned original contribution and stated hypotheses or experimental limits.
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Nina Rimsky; Nick Gabrieli; Julian Schulz; Meg Tong; Evan Hubinger; Alexander Matt Turner
Source-listed authors for the exact contribution; experimental subjects are not autonomous scientific authors. Source support and predecessor work retain separate roles. Individual effort shares and unresolved identities are not inferred.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.