The recall question from last week: there is a prior commitment, written before you look at any output, that specifies what correct means. It is not a test suite, though tests come after it. Its absence is the single most common reason a rebuilt prompt ships confident wrong answers for the second time. What is it?
Here is the answer.
The move is writing a Ground Truth Contract before you deploy or re-deploy any prompt. Not after. Not once you have seen a few outputs and formed an opinion. Before. Because the moment you look at output first, you are no longer specifying what correct means. You are rationalizing what the model produced.
A Ground Truth Contract is a written specification, scoped to a specific prompt, that defines three things: what a correct output looks like by your domain's criteria, the specific inputs for which the correct output is known in advance, and the stop-conditions that would cause you to declare the prompt failing.
Without it, you have a rebuilt prompt and an opinion. With it, you have a rebuilt prompt and a floor.
Method Deep-Dive: what a Ground Truth Contract contains
A Ground Truth Contract has three components. Each addresses a different way a rebuilt prompt can still fail in deployment.
Component 1: The evaluation criteria. What makes an output correct by your domain's definition, not by the model's general knowledge? This is the hardest component to write because it requires you to commit before seeing outputs. It forces you to articulate the conventions your team has never written down. For the changelog classifier from Issue 3: "A diff that removes a public method is breaking regardless of whether the method was documented. A diff that adds a method to an internal module is not feat." These criteria cannot come from the model. They come from the domain.
Component 2: The reference set. A set of specific inputs with their correct outputs, specified by your criteria, before you run the prompt. Not cherry-picked after observing which inputs the model handles correctly. The reference set is the ground truth: if the prompt produces the specified output on each input in the set, it passes. If it does not, it fails. The reference set does not need to be large. It needs to cover the ambiguous edge cases where the model's prior and your domain's convention diverge. A small, well-chosen set surfaces more information than a large randomly selected one.
Component 3: The stop-conditions. The specific output patterns that would cause you to stop accepting this prompt's outputs as correct and trigger a rebuild. Not "we will review it monthly." Stop-conditions specify what failure looks like in concrete terms: "If the classifier marks any diff that removes a public method as 'breaking: false', the contract is violated and the prompt must be rebuilt." Stop-conditions are what turn a Ground Truth Contract from a document into an operational gate. Without them, the criteria and reference set tell you what correct looks like but give you no trigger to act when it stops being true.
Constraint Case Study: the classifier after the rebuild
Six weeks after the BYOP rebuild from Issue 3, the changelog classifier is still misclassifying. The rate is lower. The breaking-change false negatives have dropped. But in the sprint retrospective, two diffs appeared in the release notes under the wrong type, and one diff marked "breaking: false" shipped without a migration note.
The problem is not the prompt. The BYOP rebuild was correct. The problem is that no one specified, before deploying the rebuilt prompt, what correct meant precisely enough to detect these failures.
When the failures appeared, the team had the rebuilt prompt's output. They did not have a prior record of what the correct output should have been for those specific diffs. The failure was visible in hindsight. It was not catchable in advance.
A Ground Truth Contract for the change-log classifier.
GROUND TRUTH CONTRACT — Changelog Classifier v2
Scope: TypeScript monorepo, public API surface = src/api/**
Prompt version: v2 (BYOP rebuild, post-Issue-3)
EVALUATION CRITERIA
1. A diff is "breaking: true" if any exported function, type, or interface
in src/api/** is renamed, removed, or has its signature changed such
that existing callers must update their code.
2. When uncertain whether a change is breaking, the correct output is
"breaking: true". False negatives on breaking changes are costlier
than false positives.
3. A diff is "feat" only if it adds a capability a consumer of the public
API could not previously invoke. Internal changes, performance
improvements, and test additions are not feat regardless of scope.
4. Descriptions must characterize the change from the diff. A description
that reproduces commit message text verbatim fails this contract.
REFERENCE SET (7 cases)
Case 01: diff removes exported parseConfig from src/api/config.ts
Correct: type = fix or refactor; breaking = true
Case 02: diff adds optional third param to exported formatDate
Correct: type = feat; breaking = false
Case 03: diff renames internal helper in src/utils/date.ts
Correct: type = refactor; breaking = false
Case 04: diff updates @types/node from 18 to 20
Correct: type = chore; breaking = false
Case 05: diff adds new exported function to src/api/auth.ts
Correct: type = feat; breaking = false
Case 06: diff changes return type of exported verify() from boolean
to Promise<boolean>
Correct: type = fix or feat; breaking = true
Case 07: diff removes deprecated exported constant MAX_RETRIES
Correct: type = chore; breaking = true
STOP-CONDITIONS
SC-1: Any public API removal or signature change marked "breaking: false":
contract violated; halt and rebuild.
SC-2: Any description that reproduces commit message text verbatim:
flag for review.
SC-3: Failure on cases 01, 06, or 07 (highest-consequence patterns):
rebuild required before next deployment.With this contract written before deployment, the two misclassifications from the sprint retrospective would have been detectable at the reference set stage. Case 06 covers the return-type change. Case 01 covers the removed export. The failures were predictable. They were not predicted because no one wrote down what correct meant before looking at outputs.
Vocabulary Anchor: Ground Truth Contract
Ground Truth Contract is a written specification, scoped to a specific prompt in a specific deployment context, that defines: (1) the evaluation criteria for correct output by domain standards, (2) a reference set of inputs with known-correct outputs specified before running the prompt, and (3) the stop-conditions that trigger a rebuild.
In use: "Before we redeployed the classifier, we wrote a Ground Truth Contract covering the seven edge cases that had caused misclassifications. Stop-condition SC-1 flagged a breaking-change failure in the first week of deployment."
Where it does not apply: a Ground Truth Contract specifies correctness for a fixed deployment context. It does not handle model drift over time, version changes in the underlying model, or distribution shift in the inputs. When the domain's conventions change, the contract must be updated before re-running the reference set. When the model itself changes, the reference set must be re-run against the new version. The contract governs the prompt. It does not govern the model.
Architecture Brief: what a Ground Truth Contract does not prevent
A Ground Truth Contract establishes a floor at deployment. It does not keep the floor in place.
Two things erode it. First, the model changes. A model version update alters behavior on your reference set without altering the contract. If you do not re-run the reference set against the new model version, you are running a contract against a system it was not written for.
Second, the domain's conventions change. When your team updates the definition of "breaking" or adds a new type to the classification scheme, the evaluation criteria become stale. Stale criteria mean your reference set is testing the wrong thing. You can pass the contract and still produce wrong outputs.
Catching these two failure modes requires something beyond a Ground Truth Contract: an automated pipeline that re-runs the reference set on a schedule and alerts when the model's output on reference cases deviates. That pipeline has a name. It is a Validation Suite. The Ground Truth Contract specifies what correct means. The Validation Suite enforces it over time. Issue 5 is about building one.
Closing Calibration
One thing to try this week. Take the production prompt most recently rebuilt or modified. Write its Ground Truth Contract: three to five evaluation criteria in your domain's language, five to ten reference cases with specified correct outputs, and two stop-conditions that would trigger a rebuild. Write the criteria before the next time you run the prompt.
If you find that you cannot write the criteria without looking at outputs first, that is the diagnostic result. The prompt is deployed without a prior commitment to what correct means. That is the condition that allowed second-generation misclassifications in the case study above. The contract does not need to be long. It needs to precede the output.
Next week: a Ground Truth Contract specifies correctness at a moment in time. Keeping it enforced as the model and domain change requires an automated check that runs without your intervention. The check compares live outputs to the reference set, surfaces deviations, and routes failures to a review queue. That is a Validation Suite. The most common mistake in building one is treating it as a test suite for the prompt rather than a contract-enforcement layer for the deployment. The distinction determines what you catch and what slips through.
The Constraint