依据人工标注校准模型评审器
对照人工标签计算评审一致性,分析偏差并改写评审提示词,给出留出集验证方案。
完整提示词
<context>
An LLM judge is a measuring instrument, and an uncalibrated one gives confident numbers that mean little. Raw percent agreement hides problems when classes are imbalanced: a judge that passes everything agrees 90% of the time if 90% of items are good. What matters is agreement beyond chance (Cohen's kappa for two raters, or the true-positive and true-negative rates on the failure class you care about) and where the disagreements cluster. Known judge biases: preferring longer answers, preferring the first option in pairwise comparison, favouring outputs from its own model family, leniency on vague criteria, and being swayed by confident tone. Revising the judge on the same items you measure on overfits; keep a held-out split. The human labels themselves can be inconsistent, so check human-human agreement where possible.
</context>
<task>
Calibrate this judge.
Judge prompt:
{{judge_prompt}}
Labelled results:
{{labelled_results}}
1. Compute agreement: confusion matrix, percent agreement, Cohen's kappa (or weighted kappa for ordinal scores), and the judge's true-positive and true-negative rates on the failure class. Show the arithmetic. If there are fewer than about 50 items, say the estimates are rough.
2. Split the items into a development set and a held-out test set (for example 60/40, stratified by label) if this was not done, and report which numbers come from which.
3. Read every disagreement in the development set and code why the judge differed: criterion unclear, judge too lenient, too strict, misread the input, length or tone effect, position effect (pairwise), or human label looks wrong.
4. Test for biases with the data available: agreement by output length bucket, by position in pairwise items, by which model produced the output. Say which tests the data cannot support.
5. Revise the judge prompt to address the coded causes: binary or narrow-scale criteria, definitions with pass and fail examples from the development set, a rationale before the verdict, and instructions that neutralise length and position. Suggest swapping positions and averaging for pairwise judging.
6. Give the validation plan: run the revised judge on the held-out set, the agreement target that would make it trustworthy for this use, and how often to re-check as the product changes.
7. Verdict: trust the judge for this purpose, trust it only with human spot checks, or do not use it.
</task>
<constraints>
- Do not report agreement numbers you did not compute from the given labels. If you cannot compute them reliably in your head, show the formula and the counts and say the user should run it.
- Do not tune the judge on the held-out items.
- Flag human labels that look wrong instead of silently treating them as truth.
{{> output/uncertainty}}
</constraints>
<output_format>
## Agreement
Confusion matrix, metrics and arithmetic, split by development and held-out.
## Disagreement analysis
Table: item id | human | judge | coded reason.
## Biases found
Each bias tested, the evidence, and the tests the data could not support.
## Revised judge
The full revised judge prompt.
## Validation plan
Held-out run, target agreement and re-check cadence.
## Verdict
One paragraph.
</output_format>可替换内容
- {{judge_prompt}}
- {{labelled_results}}
- {{> output/uncertainty}}
分类与标签
编程开发、模型评测、人工标注、一致性分析、偏差检测、提示词校准