Better decisions in the age of AI.

The problem

An evaluation can become less convincing when the same evidence is repeatedly used to tune a system and to demonstrate that the system works. Once developers have seen the cases, their choices may adapt—deliberately or otherwise—to those cases. A strong result on familiar evidence can therefore describe progress on the development set without establishing how the candidate performs on independent demands.

This is an evaluation-design problem. It does not imply that every reused test is useless; it means the strength of a conclusion depends on what the evidence was allowed to influence.

Development evidence and admission evidence

Development evidence helps a team find errors, compare iterations, and improve a candidate. Admission evidence asks a different question: after the candidate and evaluation conditions have been fixed, does the system meet the conditions required for a higher level of use?

The distinction is about the role of evidence, not the name of a dataset. Evidence used to guide iteration should not silently become independent confirmation of the same iteration.

Why holdout matters

A holdout is evidence reserved from development decisions and evaluated only after the candidate and relevant evaluation conditions are frozen. Its purpose is to reduce contamination between improvement and confirmation.

A holdout does not guarantee generalisation. Its value depends on case independence, scope coverage, and a defensible evaluation process. A small or unrepresentative holdout can still support only a narrow conclusion.

Why hard gates matter

In this admission framework, some conditions are intentionally non-compensatory. A high aggregate score cannot compensate for a failed critical gate, a validity problem, or a boundary violation. When a condition is essential to admission, it should remain visible as a gate rather than disappear inside a composite score.

This framework therefore treats admission as a sequence of evidence checks, not a contest to maximise one blended number.

Layered evidence

A useful mental model is:

component evidence → integrated evidence → independent holdout evidence → admission decision

Each layer answers a different question. Evidence about a component does not by itself establish that the integrated system behaves acceptably. Evidence from an integrated run does not by itself establish performance on cases withheld from development. The admission decision must state which layers were actually completed.

What this framework does not prove

The protocol behind this page defines an evaluation design, holdout requirements, and hard-gate logic. It does not establish that a holdout dataset has been completed, that a blind evaluation has been run, or that an AI system has passed admission.

Current status: protocol defined; holdout not completed; admission remains on HOLD. No admission result or system-effectiveness claim is made here.

Relation to retrieval evaluation

A retrieval evaluation framework asks: How should retrieval changes be compared under controlled conditions?

Evidence-gated evaluation asks: When is the evidence strong enough to admit a candidate to a higher validation level?

The questions are related, but they are not interchangeable. A controlled comparison can inform development; it does not automatically replace independent admission evidence.

Claim and evidence notes

问题

当同一批证据反复用于调优系统、又用于证明系统有效时，评估结论会变得不够独立。开发者已经看过测试案例，后续选择可能有意或无意地适应这些案例。因此，熟悉材料上的好结果，可以说明系统在开发证据上有所进展，却不能单独说明它面对独立需求时表现如何。

这是评估设计问题，并不意味着所有重复使用测试都毫无价值；它意味着结论的强度取决于这些证据曾影响过什么决策。

开发证据与准入证据

开发证据帮助团队发现错误、比较迭代并改进候选系统。准入证据则回答另一个问题：在候选系统及评价条件固定之后，它是否满足进入更高使用或验证层级的条件？

两者的区别在于证据承担的角色，而不只是数据集名称。用于指导迭代的证据，不应不加说明地变成对同一轮迭代的独立确认。

为什么需要 Holdout

Holdout 是一组不参与开发决策、只有在候选系统和相关评估条件冻结后才用于评价的证据。它的作用是降低“改进系统”与“确认系统”之间的证据污染。

Holdout 不保证泛化。其价值仍取决于案例独立性、范围覆盖和评价过程是否站得住脚。规模很小或缺乏代表性的 holdout，仍只能支撑有限结论。

为什么需要硬门槛

在这套准入框架中，有些条件被有意设计为不可补偿。较高的综合分不能抵消关键门槛失败、评价效度问题或边界违规。如果某项条件是准入所必需的，它就应作为清晰可见的门槛，而不是被综合分掩盖。

因此，这个框架把准入视为一组分层证据检查，而不是最大化一个混合分数。

分层证据

可以用下面的心智模型理解：

组件证据 → 集成证据 → 独立 Holdout 证据 → 准入决策

每一层回答不同问题。组件表现不能单独证明集成系统的行为可接受；集成运行的证据也不能单独证明系统在开发阶段未见案例上的表现。准入结论必须说明哪些层级确实完成了。

这个框架不能证明什么

本页所依据的协议定义了评估设计、holdout 要求和硬门槛逻辑。它并未证明 holdout 数据集已经完成、盲评已经执行，或某个 AI 系统已经通过准入。

当前状态：协议已定义 → Holdout 尚未完成 → 准入仍为 HOLD。本页不声称已经取得准入结果，也不声称系统效果已获证明。

与检索评估的关系

检索评估框架回答：如何在受控条件下比较检索改动？

证据门控评估回答：证据何时足以支持候选系统进入更高验证层级？

两者相关，但不能互相替代。受控比较可以支持开发判断，却不会自动取代独立准入证据。

主张与证据说明

The shift

AI systems can increasingly execute tasks that previously required direct human effort. That change raises a question beyond task replacement:

As AI takes on more execution, which forms of agency should remain human?

Here, agency means the ability and authority to shape goals, set constraints, judge whether work is acceptable, and take responsibility for consequential commitments. This page offers an organising framework for thinking about those roles; it is not a validated general theory of organisational design.

Execution is not decision authority

Delegating execution does not automatically delegate the authority to decide what should be done, what risks are acceptable, or who is accountable when an outcome matters.

An AI system may draft, search, transform, or route information. A human or organisation still has to define the objective, establish boundaries, decide how outputs will be checked, and determine which commitments require human ownership. The appropriate allocation depends on the task, its uncertainty, and the consequences of error.

Four dimensions of retained human agency

The source synthesis distinguishes four human roles:

1. Goal setting — defining objectives and priorities.
2. Constraint design — specifying permissions, boundaries, risk limits, and acceptance criteria.
3. Verification architecture — deciding what evidence and checks are needed, and when exceptions should be escalated.
4. Accountability and commitment — owning consequential decisions, especially when a choice is difficult to reverse.

These are lenses for examining a workflow, not a universal sequence or a claim that every decision must remain human. Their boundaries can overlap, and actual allocations require context-specific design.

What should remain human?

A useful design question is not “Can this task be automated?” alone. Ask also:

• Who chooses the objective and resolves conflicts between objectives?
• Who sets the boundary of acceptable action?
• What evidence is enough to trust the output?
• Which errors require escalation?
• Who can pause, reverse, or own a consequential commitment?

As execution moves toward AI, accountability does not move with it by default. That is a governance choice that should be made explicitly.

Why harnesses matter

A harness is the surrounding structure that constrains, routes, and verifies AI execution. It can make a workflow more legible by stating its inputs, limits, checks, and escalation points.

A harness does not make uncertain goals clear by itself, guarantee that verification is reliable, or remove human responsibility. It is a way to organise delegation and oversight, not a substitute for judgment.

Open questions

The framework motivates questions that remain open rather than established conclusions:

• Under what conditions does redistributing execution change decision quality?
• When does a harness reduce coordination burden, and when does it add overhead?
• How does the right allocation change as task uncertainty or reversibility changes?
• Which forms of verification can be delegated without weakening accountability?

No productivity, scalability, cost-saving, or decision-quality effect is claimed here.

Relation to Human-in-the-Loop

Human-in-the-Loop asks when human intervention or approval is needed at a decision point.

Agency redistribution asks how authority and responsibility are allocated across a workflow, including before and after that intervention point.

The first focuses on an intervention mechanism; the second is a broader lens for mapping decision roles.

Claim and evidence notes

转变

AI 系统正在承担越来越多过去需要人直接完成的任务。这带来的问题不只是岗位替代：

当 AI 承担更多执行时，哪些形式的能动性应当留在人类手中？

这里的 Agency（能动性）指影响目标、设定约束、判断工作是否可接受，并对重要承诺承担责任的能力与权力。本页提供一个组织这些问题的分析框架；它不是已经验证的一般组织设计理论。

执行权不等于决策权

把执行任务交给 AI，并不会自动把“决定做什么”“哪些风险可以接受”或“重要结果由谁负责”的权力一并交出去。

AI 可以起草、检索、转换或路由信息；人或组织仍需定义目标、设置边界、决定如何检查输出，并判断哪些承诺需要由人负责。合理分工取决于任务、不确定性和错误后果。

人类 Agency 的四个维度

来源综合将人的角色组织为四个维度：

1. 目标设定：定义目标与优先级。
2. 约束设计：说明权限、边界、风险限制和验收标准。
3. 验证架构：决定需要哪些证据与检查，以及何时升级异常。
4. 责任与承诺：对重要决策负责，尤其是难以逆转的选择。

这些层面是分析工作流的视角，不是普遍适用的固定顺序，也不意味着每项决策都必须由人完成。它们可能彼此交叠，实际分配需要结合情境设计。

哪些权力应当留给人？

设计时不能只问“这项任务能否自动化”，还应问：

• 谁选择目标，并处理目标之间的冲突？
• 谁划定可接受行动的边界？
• 什么证据足以支持对输出的信任？
• 哪些错误必须升级处理？
• 谁能暂停、撤回或承担重要承诺？

执行权向 AI 转移时，责任不会自动随之转移。这是一项需要明确作出的治理选择。

Harness 为什么重要

Harness 是围绕 AI 执行建立的结构，用来约束、路由和验证任务。它可以通过明确输入、限制、检查与升级节点，使工作流更清晰。

Harness 本身不会自动澄清不确定目标，不保证验证可靠，也不会消除人的责任。它用于组织委派与监督，不能替代判断。

尚待回答的问题

这个框架引出一些仍待研究的问题，而不是已经成立的结论：

• 在什么条件下，重新分配执行权会改变决策质量？
• Harness 何时降低协调负担，何时反而增加开销？
• 任务不确定性或可逆性变化时，合适的分工如何变化？
• 哪些验证工作可以委派而不削弱责任边界？

本页不声称生产率、组织扩展性、成本节约或决策质量已经改善。

与 Human-in-the-Loop 的关系

Human-in-the-Loop 回答：在某个决策点，何时需要人工介入或批准？

Agency 再分配回答：一项工作流中，决策权和责任如何分布？这也包括介入点之前与之后的分工。

前者关注介入机制；后者提供一张更广的决策角色地图。

主张与证据说明

One vivid case, or a set of comparable cases?

A plan can look carefully reasoned. The team explains its assumptions, lays out the steps, and describes why this situation is different. Before accepting that account, it helps to ask a quieter question: how often have comparable plans reached the intended outcome?

That question moves judgment from the case in front of us to the broader set of cases it belongs to. A base rate provides context from that set; case-specific information helps us decide whether this case differs in a relevant way. They play different roles, and neither automatically replaces the other.

Start with the reference class, then return to the case

Kahneman and Tversky’s research on prediction discussed the representativeness heuristic: people may form predictions from how much a case resembles a familiar prototype, without adequately considering prior probabilities or the reliability of evidence. Their later review of judgment under uncertainty described representativeness, availability, and anchoring as heuristics that are often useful but can also produce systematic biases.[1][2]

This does not mean that base rates are always right or that case details do not matter. It suggests a check: when a story feels especially vivid, coherent, or persuasive, ask what reference group makes it representative. Then examine whether the supporting evidence is reliable and genuinely diagnostic.

Put both kinds of information on the table

Prediction can involve information about an individual case and information about a distribution of cases. In their discussion of intuitive prediction and corrective procedures, Kahneman and Tversky distinguish data about a singular case from distributional data.[3]

In the narrower domain of project forecasting, reference-class forecasting offers an outside view: examine the actual outcomes of comparable projects, then return to the project at hand. Flyvbjerg’s account focuses on project forecasting, particularly cost risk. The point used here is the practice of consulting relevant past cases; the method’s effects are not generalized to every personal or organizational judgment.[4]

In practice, ask:

1. What kind of judgment is being made?
2. Which past cases are sufficiently comparable on the characteristics and outcome conditions relevant to this judgment?
3. What context does their outcome distribution provide?
4. What information about the current case is credible, relevant, and diagnostic?
5. Are the differences from the reference cases strong enough to justify a different judgment?

This set of questions is the author’s working synthesis, not a verbatim procedure from a single study.

Choosing a reference class is itself a judgment

Selecting a comparison group is not mechanical. A class that is too broad may combine unlike cases; one that is too narrow may leave too little data or include only cases that fit an expectation. Changes in organizational, technological, or institutional conditions can also limit how well an older distribution applies.

A base rate is therefore an input whose source, scope, and limitations need to be explained, not an answer that makes the decision for us. Case-specific evidence deserves the same scrutiny: Is its source reliable? Is it relevant? Is it strong enough to change the judgment?

Record the judgment and leave room to review it

One practical discipline is to record, before the outcome is known, the reference class, the base-rate information used, the key case-specific evidence, the forecast, and its uncertainties. Review the judgment against the outcome later. Keeping a record of forecasts and outcomes is a practical recommendation from the author; it should not be mistaken for a universally validated intervention established by the sources above.

Calibration here primarily means making a forecast traceable and comparing it with the eventual outcome. Unless calibration metrics are actually computed, the term does not imply formal probabilistic calibration.

AI can help organize comparable cases, make assumptions visible, or generate alternative explanations. A fluent case narrative still needs checks on its sources, relevance, and scope. This is a conceptual connection to the judgment process; it does not claim that AI causes base-rate neglect or that AI use necessarily improves forecasting.

Limits

• The cases found may not form an appropriate reference class; comparability needs an argument.
• Small samples, data quality, and selection methods affect how a distribution should be interpreted.
• Structural change or a genuinely rare new situation may reduce the relevance of historical comparisons.
• This essay concerns prediction and judgment calibration. It does not infer causality or provide a statistical model or individualized probability advice.
• How base rates and case-specific evidence should be weighed depends on the question, evidence quality, and context.

Claim and evidence boundaries

References

1. Kahneman, D., & Tversky, A. (1973). “On the Psychology of Prediction.” Psychological Review, 80(4), 237–251. https://doi.org/10.1037/h0034747
2. Tversky, A., & Kahneman, D. (1974). “Judgment under Uncertainty: Heuristics and Biases.” Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124. PubMed record
3. Kahneman, D., & Tversky, A. (1982). “Intuitive Prediction: Biases and Corrective Procedures.” In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under Uncertainty: Heuristics and Biases, 414–421. Cambridge University Press. https://doi.org/10.1017/CBO9780511809477.031
4. Flyvbjerg, B. (2006). “From Nobel Prize to Project Management: Getting Risks Right.” Project Management Journal, 37(3), 5–15. https://doi.org/10.1177/875697280603700302

一个具体案例，还是一组相似案例？

一个计划看起来很周全：团队解释了关键假设，列出执行步骤，也说明了为什么这次情况不同。这样的叙述可能很有说服力。但在接受它之前，还可以问一个更朴素的问题：过去相似的计划通常走到了哪一步？

这个问题把判断从眼前个案带到它所属的一组案例。基准率提供这组案例的背景；个案信息则帮助我们判断这一次是否确有重要差异。二者承担不同角色，不能由其中一方自动取代另一方。

先看参照类，再回到个案

Kahneman 与 Tversky 对预测判断的研究讨论了代表性启发式：人们可能依据一个案例与某种典型情形有多相似来形成预测，而没有充分考虑先验概率或证据可靠性。后续关于不确定情境下判断的综述也指出，代表性、可得性和锚定等启发式通常有用，却可能带来系统性偏差。[1][2]

这并不意味着基准率永远正确，也不意味着个案细节都不重要。它提示的是一种检查方式：当一个故事格外鲜明、连贯或令人信服时，先问它相对于什么参照群体显得典型；再检查支持它的证据是否可靠、是否真正具有区分力。

把两类信息放在同一张桌上

预测可以同时面对个案层面的信息与分布层面的信息。Kahneman 与 Tversky 对直觉预测与校正程序的讨论明确区分了关于单一案例的数据和关于一组案例分布的数据。[3]

在项目预测这一较窄的领域，reference-class forecasting（参考类预测）提供了一个外部视角：先观察可比项目的实际结果，再回到当前项目评估预测。Flyvbjerg 对这一方法的论述聚焦于项目预测，尤其是项目成本风险；这里借用的是它“先看相关案例的实际表现”的思路，不把该方法的效果外推到所有个人或组织判断。[4]

实际使用时，可以依次问：

1. 这次判断属于哪一类问题？
2. 哪些既往案例在与当前判断相关的关键特征和结果条件上足够可比，可以作为参照？
3. 这些案例的结果分布提供了什么背景？
4. 当前个案有哪些可信、相关且有区分力的信息？
5. 当前情形与参照案例的差异，是否足以支持不同的判断？

这组问题是本文作者的工作性综合，不是对某一项研究程序的逐字转述。

参考类本身也需要判断

选择参照群体不是机械步骤。范围太宽，案例之间可能并不相似；范围太窄，样本可能不足，或者只挑进了与预期相符的案例。组织、技术或制度条件变化时，旧分布的适用性也需要重新评估。

所以，基准率不是替我们作决定的答案，而是一项需要说明来源、范围和局限的输入。个案证据也需要接受同样的检查：来源是否可靠，证据是否相关，它是否足以改变原有判断？

记录判断，留出复盘空间

一种实用做法是，在结果出现前记录：所选参照类、判断所依据的基准信息、当前个案的关键证据、预测及其不确定之处。之后再对照结果复盘。持续记录预测与结果，是本文提出的校准实践建议；它不应被误读为上述文献已验证的通用干预效果。

这里的“校准”主要指把事前判断与事后结果进行可追溯比较；除非实际计算校准指标，否则不表示统计意义上的概率校准。

AI 可以帮助整理可比案例、显化假设或生成替代解释。但一个流畅的个案故事仍需经过来源、相关性与适用范围的检查。这里讨论的是判断流程上的关联，并不主张 AI 会导致基准率忽视，也不声称使用 AI 必然改善预测。

适用边界

• 找到的案例未必构成合适的参考类；可比性需要论证。
• 小样本、数据质量和选择方式会影响分布信息的解释。
• 结构变化或真正罕见的新情形可能削弱历史参照的适用性。
• 本文讨论预测与判断校准，不据此推断因果关系，也不提供统计模型或个体化概率建议。
• 基准率与个案证据如何权衡，仍取决于问题、证据质量与情境。

Claim 与证据边界

参考资料

1. Kahneman, D., & Tversky, A. (1973). “On the Psychology of Prediction.” Psychological Review, 80(4), 237–251. https://doi.org/10.1037/h0034747
2. Tversky, A., & Kahneman, D. (1974). “Judgment under Uncertainty: Heuristics and Biases.” Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124. PubMed record
3. Kahneman, D., & Tversky, A. (1982). “Intuitive Prediction: Biases and Corrective Procedures.” In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under Uncertainty: Heuristics and Biases, 414–421. Cambridge University Press. https://doi.org/10.1017/CBO9780511809477.031
4. Flyvbjerg, B. (2006). “From Nobel Prize to Project Management: Getting Risks Right.” Project Management Journal, 37(3), 5–15. https://doi.org/10.1177/875697280603700302