In Issue 4, you wrote a Ground Truth Contract for your changelog classifier. You specified the input type, success criteria, four pass/fail conditions, and the format the output must match. You tested it against ten reference cases. It passed. You deployed.
Three months later, the model version your hosting provider runs has been updated. Your team extended the schema the classifier reads. Two new change types appeared in production that were not in your reference set. The contract you wrote is still in the system prompt. It is no longer being enforced. You will not know until a failure is visible in the output.
That is not a prompt failure. It is a validation failure: the Ground Truth Contract held at deployment time, and no mechanism kept it holding as conditions evolved.
The Validation Suite is that mechanism.
Method Deep-Dive: test suite vs. Validation Suite
A Validation Suite is not a test suite. The distinction matters.
A test suite answers: does this prompt produce correct output on these cases right now? You run it before deployment. It is a point-in-time check against a fixed input set.
A Validation Suite answers: does this deployment remain correct as conditions change? It runs continuously against live outputs, compares results to the Ground Truth Contract, and surfaces deviations without manual intervention.
A Validation Suite has four required components.
Component 1: Reference Set. A fixed collection of (input, expected output, evaluation criteria) triples drawn from real production inputs, not synthetic ones. The reference set must include canonical cases (inputs the prompt handles correctly by design), edge cases (inputs near the boundary of the prompt's stated scope), and known-failure cases (inputs the prompt previously failed on, with the correct output now documented). A reference set built entirely from canonical cases will not catch boundary drift.
Component 2: Evaluation Criteria. The exact conditions from the Ground Truth Contract, expressed as evaluable statements. Not "the output is correct" but "the output contains a severity field with one of three valid values" and "the output does not include content from the field labelled user_comment." Criteria that cannot be expressed as evaluable statements are not yet precise enough to enforce automatically.
Component 3: Evaluation Mechanism. The method that compares live output to the reference set. Two options exist: deterministic checks (schema validation, field-presence checks, regex) and LLM-as-Judge (a separate model call that evaluates whether the live output satisfies the Ground Truth Contract criteria). Deterministic checks are faster and cheaper. LLM-as-Judge handles criteria too semantic for pattern matching. Production-grade Validation Suites use both: deterministic checks gate first, LLM-as-Judge evaluates what passes the gate.
Component 4: Routing Policy. What happens when a check fails. A Validation Suite with no routing policy produces a log of failures that accumulates without action. The routing policy has three destinations: auto-pass (both checks confirm), review queue (one check fails or is uncertain), and immediate alert (a known-failure case now passes, which may indicate unintended drift, or a canonical case fails, which is a production regression).
Constraint Case Study: automated enforcement for the changelog classifier
The changelog classifier from Issue 4 categorised changes from commit metadata: breaking changes, deprecations, and routine updates. The Ground Truth Contract defined correctness as a structured JSON output with four fields (change_type, severity, scope, summary), each with enumerated valid values.
The following is a minimal Validation Suite for that classifier. Deterministic schema checks gate first. LLM-as-Judge evaluates the summary field, which schema validation cannot reach.
python
import json
import openai
REFERENCE_SET = [
{
"input": {"commit_msg": "BREAKING: removed /v1/users endpoint", "diff_lines": 47},
"expected": {"change_type": "breaking", "severity": "high", "scope": "api"},
"summary_criteria": (
"summary must reference endpoint removal; "
"must not contain speculation about downstream impact"
)
},
{
"input": {"commit_msg": "fix: correct null handling in parser", "diff_lines": 3},
"expected": {"change_type": "routine", "severity": "low", "scope": "internal"},
"summary_criteria": (
"summary must identify the bug corrected; "
"must not escalate severity"
)
},
# known-failure case: classifier previously misclassified deprecation notices as breaking
{
"input": {
"commit_msg": "deprecated: /v2/legacy route will be removed in Q4",
"diff_lines": 2
},
"expected": {"change_type": "deprecation", "severity": "medium", "scope": "api"},
"summary_criteria": (
"summary must reference deprecation timeline; "
"change_type must not be breaking"
)
}
]
VALID_VALUES = {
"change_type": {"breaking", "deprecation", "routine"},
"severity": {"high", "medium", "low"},
"scope": {"api", "internal", "config", "docs"}
}
def validate_schema(output: dict) -> list[str]:
return [
f"{field}: '{output.get(field)}' not in {valid}"
for field, valid in VALID_VALUES.items()
if output.get(field) not in valid
]
def llm_judge(output: dict, criteria: str) -> dict:
prompt = (
f"Evaluate whether this classifier output meets the stated criteria.\n\n"
f"Output: {json.dumps(output)}\n"
f"Criteria: {criteria}\n\n"
f'Return JSON: {{"pass": true/false, "reason": "one sentence"}}'
)
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)
def run_validation_suite(classifier_fn) -> list[dict]:
results = []
for case in REFERENCE_SET:
output = classifier_fn(case["input"])
schema_failures = validate_schema(output)
if schema_failures:
results.append({
"status": "FAIL", "stage": "schema",
"failures": schema_failures, "case": case
})
continue
field_failures = [
f for f, v in case["expected"].items()
if output.get(f) != v
]
if field_failures:
results.append({
"status": "REVIEW", "stage": "field_match",
"failures": field_failures, "case": case
})
continue
judge = llm_judge(output, case["summary_criteria"])
if not judge["pass"]:
results.append({
"status": "REVIEW", "stage": "llm_judge",
"reason": judge["reason"], "case": case
})
else:
results.append({"status": "PASS", "case": case})
return resultsThree routing outcomes: PASS (both checks clear), REVIEW (one check uncertain or failed), FAIL (schema invalid).
The known-failure case is the most important entry in the reference set. If the classifier now produces change_type: breaking for a deprecation notice, that is not a pass. The reference set documents that failure pattern explicitly so the Validation Suite catches regression in either direction, not just one.
Vocabulary Anchor: Validation Suite
The Validation Suite is a structured set of reference cases, evaluation criteria, and an evaluation mechanism that runs continuously against a deployed prompt's live outputs to enforce the Ground Truth Contract as conditions change. A Validation Suite is not a test suite: it does not run once before deployment, it does not answer whether the prompt is correct right now, and it does not pass or fail the prompt as a whole. It runs in production, surfaces deviations from the contract, and routes failures for review. A prompt can pass its test suite and fail its Validation Suite simultaneously.
In use: "The model update in Q3 did not break schema validation, but it shifted how the classifier worded summaries for edge-scope changes. The Validation Suite caught three LLM-as-Judge failures in the first 48 hours. The test suite would not have caught any of them."
Where it does not apply: a Validation Suite enforces a Ground Truth Contract that has already been written. It cannot compensate for an absent or imprecise contract. If the evaluation criteria are not specific enough for deterministic checks or for an LLM-as-Judge system prompt to return a consistent verdict, the Validation Suite will produce noisy, unreliable routing decisions. The Ground Truth Contract must be precise before the Validation Suite can be useful. Attempting to build a Validation Suite before a Ground Truth Contract is the most common sequencing error in production AI validation.
Architecture Brief: the sequence is not optional
The Ground Truth Contract, Validation Suite, and LLM-as-Judge are three components of a single workflow, not three independent options.
The Ground Truth Contract makes correctness explicit. The Validation Suite enforces it continuously. LLM-as-Judge evaluates the criteria the contract defines that schema checks cannot reach.
Skip the Ground Truth Contract and the Validation Suite has nothing to enforce. Build the Validation Suite without LLM-as-Judge and semantic failures pass the gate. Use LLM-as-Judge without a Ground Truth Contract and you are asking a model to judge correctness you have not yet defined.
The reason most AI agent deployments degrade slowly over time is not that the initial prompt was wrong. It is that no mechanism detected when it stopped being right.
Closing Calibration
One thing to try this week. Pick one deployed prompt and write three reference cases for it: one canonical case (an input the prompt handles correctly by design), one edge case (an input near the boundary of its stated scope), and one known-failure case (an input it previously failed on, with the correct output now documented).
That is the seed reference set for a Validation Suite.
You do not need to run the suite yet. Writing the three cases will show you whether your Ground Truth Contract is precise enough to derive evaluation criteria from. If you cannot write the expected output for the edge case, the contract has a gap. That gap is what the Validation Suite would not have been able to catch.
Next week: the Validation Suite catches a failure. The schema check passes. LLM-as-Judge returns REVIEW. The prompt specification is unchanged. The model version is unchanged. But the classifier is wrong. The source of the failure is not in anything you wrote. The diagnostic for this type of failure is different from the BYOP diagnostic. It starts by asking what changed in the domain, not in the prompt.
The Constraint