Why two impressive percentages may disagree
Suppose two reviewers give the same pair of products different marks. One checks ten short answers; the other checks twenty longer ones. Even if both calculate percentages correctly, the marks can disagree because the tests differ. That is a measurement problem before it is a product-quality problem. A shared test needs to say which inputs, resources and rules it uses.
The contribution is a common evaluation procedure
HarmBench standardizes safety stress-test behaviors, generation settings and evaluation, including held-out cases and classifiers. Its original non-copyright criterion can count an attempted behavior even when the output is incorrect. That measures a different question from whether a response would actually work. Keeping those purposes separate is essential when reading a ranking.
Try a harmless classroom comparison
This invented example uses arithmetic questions. Model A answers seven of ten correctly. Model B answers fourteen of twenty correctly. Both have seventy percent accuracy, but we still need to know whether they faced comparable questions. Next, test both on the same ten new questions with the same allowed answer length and number of attempts. If B now answers eight correctly, its result is eighty percent on that particular shared test. This does not prove it wins on every future question.
Keep the answer key away from tuning
A held-out test is a set of questions reserved for evaluation rather than repeated practice. If a developer keeps changing a system after reading its exact test mistakes, the final mark partly measures adaptation to that test. An unseen set helps check transfer. Even then, a small or biased test can miss important behavior. The result should retain its question set, judging rule and system version instead of becoming a timeless claim of superiority.
What you can add without expensive experiments
Choose two published evaluation tables and make a comparison contract: task set, split, target versions, generation limits, judge, and success definition. Mark every mismatch. You can then explain which comparisons are direct and which remain uncertain. A complete table is not required for a useful contribution: one source-backed discovery that the two studies use different success definitions can prevent a misleading headline or guide a better follow-up.
Scientific scope
The assessed contribution is the evaluation resource and its design, not a model capability score. Results depend on held-out behaviors, classifiers, generated length and attack budgets. For non-copyright cases, the original success criterion includes attempted behavior even when the output is incorrect. It is not exhaustive harm coverage, verified harmful capability or a deployment safety certificate.
Read the original work
Sources and reading locations for this explanation
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Pinned original contribution and stated hypotheses or experimental limits.
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Mantas Mazeika; Long Phan; Xuwang Yin; Andy Zou; Zifan Wang; Norman Mu; Elham Sakhaee; Nathaniel Li; Steven Basart; Bo Li; David Forsyth; Dan Hendrycks
Source-listed authors for the exact contribution; experimental subjects are not autonomous scientific authors. Source support and predecessor work retain separate roles. Individual effort shares and unresolved identities are not inferred.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.