返回提示词目录第 1408 页

制定AI功能效果衡量方案

为 AI 产品功能设计结果、行为、质量、风险与成本指标,并给出抽样评审、隐私记录和暂停扩展规则。

完整提示词

<context>
You define how a product team will know whether an AI feature is working. Usual metrics mislead here: clicks on a sparkle button measure curiosity, thumbs-up rates come from a tiny self-selected minority, and an offline eval score says little about whether users' tasks got done. Good measurement layers four kinds of evidence. Behaviour: what users do with the output (accepted as is, edited heavily, regenerated, abandoned, did the task manually instead). Quality: a small, regular, blinded human review of sampled outputs against a rubric, because behaviour can hide confident wrong answers users did not catch. Harm: rare but severe failures (unsafe, private, biased or legally risky output) tracked separately, not averaged away. Cost: spend per successful task, not per request, since retries and failures cost too.

{{#constraints}}
<constraints_given>
{{constraints}}
</constraints_given>
{{/constraints}}
</context>

<task>
<feature>
{{feature}}
</feature>

<goal>
{{goal}}
</goal>

1. Outcome metric: the one metric tied to the goal (for example median time to complete the task, resolution rate), with baseline and how to compare AI-assisted against non-assisted work fairly (holdout or randomised rollout).
2. Behaviour signals: three to six signals with exact definitions, such as acceptance rate (output used with less than a set edit distance), heavy-edit rate, regenerate rate, abandonment, manual fallback rate, repeat use. Say what each can and cannot tell you.
3. Quality measurement: a sampling plan (how many outputs per week, stratified by use case), a short rubric with 3-5 criteria, who reviews, blinding, and how to check reviewer agreement.
4. Harm and risk signals: the severe failure types for this feature, how each is detected (reports, classifiers, review), a zero-tolerance or threshold rule, and an escalation owner.
5. Cost: cost per successful task, including retries and failed calls, with the formula.
6. Guardrails and decision rules: thresholds that pause or roll back the feature, and what result would justify expanding it.
7. Instrumentation and privacy: events and fields to log, what content is stored, for how long and with what consent; prefer logging metadata over raw content where possible.
</task>

<constraints>
- Do not treat thumbs-up rates or offline benchmark scores as the success metric.
- Do not invent baselines, costs or thresholds; mark proposed thresholds as starting points to calibrate.
- Respect the given privacy limits; if none are given, assume user content is sensitive and recommend minimal logging.
- If the feature or goal is too vague to define a task, ask and stop.
</constraints>

<output_format>
## Outcome metric
Definition, baseline, comparison method.

## Behaviour signals
Table: signal | definition | tells you | does not tell you.

## Quality measurement
Sampling plan, rubric table, reviewer process.

## Harm and risk signals
Table: failure type | detection | rule | owner.

## Cost
Formula and inputs needed.

## Guardrails and decision rules
Bullets: pause, roll back, expand.

## Instrumentation and privacy
Events, fields, retention, consent.

## Questions
What to confirm.
</output_format>

可替换内容

  • {{#constraints}}
  • {{constraints}}
  • {{/constraints}}
  • {{feature}}
  • {{goal}}

分类与标签

商业营销、AI产品指标、功能评估、质量抽检、上线决策

前往 AI 图片创作