In Issue 6, you built a Validation Suite for the changelog classifier. LLM-as-Judge and schema checks run continuously against live outputs. The routing policy sends failures to the review queue and fires an immediate alert for production regressions.
Six weeks after deployment, the alert fires. Three reference cases are returning REVIEW that used to return PASS.
You open the cases. The schema is valid. The field values are correct. Only the LLM-as-Judge evaluation is failing. For all three cases, the reason is the same: "summary contains forward-looking migration guidance inconsistent with deprecation criteria."
You check the prompt. Nothing has changed since you deployed it.
You check the model version. No update in the past two weeks.
You check the reference set. The failing cases are the deprecation notices: the same category that was a known-failure case when you built the contract in Issue 4.
Then you look at the actual commit messages that produced those outputs. Six weeks ago, the team adopted a new deprecation convention. Where they used to write "deprecated: /v2/legacy route will be removed in Q4," they now write "deprecated: /v2/legacy route will be retired Q4; teams should migrate to /v3/route by [date]." The change was announced in a Slack thread you were not in.
The commit message convention changed. The Ground Truth Contract did not.
This is not a prompt failure. It is a Semantic Drift Vector.
Method Deep-Dive: what the Semantic Drift Vector is, and is not
The Semantic Drift Vector is the divergence between the specification a prompt was written against and the domain it operates in, as the domain evolves after deployment.
It has a direction: drift moves from the specification toward the domain's current state. It has a magnitude: the gap between what the Ground Truth Contract expects and what live inputs now produce. And it is not caused by the prompt being wrong. The prompt was correct when you wrote it. The domain changed under a specification that did not follow.
Three channels produce Semantic Drift in production.
Channel 1: Vocabulary drift. The terms a domain uses to describe its own concepts shift. "Breaking change" in your codebase now includes something your original definition did not cover. "Deprecation" now carries migration guidance that your evaluation criteria did not anticipate. The words in the prompt are the same; the words in the inputs have evolved.
Channel 2: Convention drift. The team changes how it structures artifacts: commit messages, tickets, support cases, reports. The format the evaluation criteria assumed no longer matches the format the inputs arrive in. Schema validation passes because the field types are correct. LLM-as-Judge flags the structural change because the semantic pattern is different.
Channel 3: Scope drift. The domain's edge cases migrate. What was an edge case when you built the reference set is now a canonical case. What was canonical is now rare or absent. The reference set no longer represents the distribution of live inputs.
All three produce the same symptom: a Validation Suite that fires on a prompt that has not been changed. The failure is real. The cause is not the prompt.
Constraint Case Study: the diagnostic in practice
The changelog classifier's Validation Suite caught three REVIEW cases. The prompt is unchanged. Here is the diagnostic sequence.
Step 1: Check the prompt for changes. Compare the current system prompt to the version active at the last known-good Validation Suite run. No diff. The prompt is not the cause.
Step 2: Check the model version. Confirm the model version has not changed since the last known-good run. Unchanged. Model drift is not the cause.
Step 3: Check the Ground Truth Contract against live inputs. Pull five recent live inputs from the category that is failing (deprecation notices). Compare them to the input structure in the reference set. The structure is different: the new deprecation format includes migration guidance that was absent when you wrote the evaluation criteria.
That is the cause. The evaluation criteria says "summary must reference deprecation timeline; change_type must not be breaking." The new format produces summaries that reference the deprecation timeline but also include migration guidance. The LLM-as-Judge reads the migration guidance as potential breaking-change intent and returns REVIEW.
The fix: update the Ground Truth Contract evaluation criteria, not the prompt.
Old criteria: "summary must reference deprecation timeline; change_type must not be breaking."
New criteria: "summary must reference deprecation timeline; change_type must not be breaking; migration guidance is acceptable in deprecation summaries and does not indicate breaking intent."
Then update the reference set: replace the stored input with a current example of the new deprecation format, and verify the expected output against the updated criteria before re-running the suite.
The prompt does not change. The prompt was correct. What changed is the specification that the Validation Suite enforces.
python
Ground Truth Contract update: deprecation criteria patch
Run after identifying Semantic Drift Vector in deprecation channel
OLD_CRITERIA = (
"summary must reference deprecation timeline; "
"change_type must not be breaking"
)
NEW_CRITERIA = (
"summary must reference deprecation timeline; "
"change_type must not be breaking; "
"migration guidance is acceptable in deprecation summaries "
"and does not indicate breaking intent"
)
Update REFERENCE_SET entry for the deprecation case
Verify: new criteria + new input format → PASS before re-deploying suite
This is the minimum viable Semantic Drift patch: one criteria update, one reference set entry updated, one verification run. The suite re-runs and the failing cases return PASS.
Vocabulary Anchor: Semantic Drift Vector
The Semantic Drift Vector is the divergence between the specification a prompt was written against and the domain it operates in, after the domain has evolved. It is a vector (not a gap) because it has a direction (toward the domain's current state) and a magnitude (the distance between what the contract expects and what live inputs produce).
The Semantic Drift Vector is distinct from prompt failure (the prompt specification is wrong) and model drift (the model's behavior has changed). Semantic Drift is a property of the relationship between a specification and a domain. The specification was correct at time of writing. The domain moved.
In use: "The Validation Suite failures were not a prompt regression. The team's commit message conventions changed in week six. The contract's evaluation criteria had not been updated. We patched the criteria and re-ran the suite. PASS."
Where it does not apply: when the prompt itself is wrong: when the specification was never correct for the domain. The BYOP diagnostic (Issue 3) is the right tool for that failure. The Semantic Drift Vector applies only when the prompt was correct and the domain changed after deployment. Confusing the two leads to rebuilding correct prompts instead of updating stale contracts. Try this diagnostic tool.
Architecture Brief: the three-way diagnostic
When a Validation Suite fires, there are three possible causes. The diagnostic question before touching the prompt is always: what changed?
Cause 1: Prompt regression. The prompt was modified after the last known-good run. Check the diff. If the prompt changed, run the full reference set against the previous prompt version. If it passes, the change introduced the failure. Revert or fix the change. This is a standard regression: the prompt is the cause.
Cause 2: Semantic Drift Vector. The prompt has not changed. The domain has. Commit message conventions, document formats, team terminology, input structure: something in the domain evolved after the Ground Truth Contract was written. The fix is to update the contract and the reference set, not the prompt. The prompt remains correct for the domain as it was; the contract must follow the domain as it is.
Cause 3: Model drift. The prompt has not changed. The domain has not changed. The model version has updated. The model's behavior on your reference cases has shifted. Run the full reference set against the previous model version if available. If behavior diverges, assess whether the prompt needs adjustment or whether the evaluation criteria need to account for the model's updated defaults. This is model drift: the specification may need a targeted update to compensate for changed model behavior.
The three causes require three different interventions. Applying the wrong fix to the wrong cause resets the clock: a contract update will not fix a prompt regression; a prompt rebuild will not fix Semantic Drift.
Most teams default to rebuilding the prompt because that is the most visible lever. For Semantic Drift, it is the wrong lever. It is also the most expensive: rebuilding a correct prompt destroys the investment in the original specification, requires a new Ground Truth Contract, and resets the Validation Suite's baseline. Updating the contract takes minutes.
The diagnostic runs before any remediation begins. What changed?
Try our free Prompt Diagnostic Engine GPT
Closing Calibration
One thing to check this week. Look at your most recent Validation Suite failure. If you have not built one yet, use your most recent prompt failure that appeared in production after a period of stable outputs.
Ask three questions in order: Did the prompt change? Did the model version change? Did the domain change?
If the first two are no and the third is yes, you are looking at a Semantic Drift Vector. The fix is a contract update, not a prompt rebuild.
If you cannot answer the third question without checking, that is useful information. It tells you that your team's domain evolution is not being tracked in a place that is visible at the moment the Validation Suite fires. The fix for that is a documented change log for your Ground Truth Contracts: a record of when the domain changed and what the contract update was, with the date. A contract without a change log is correct at one moment in time. A contract with a change log is defensible across the deployment period.
Next week: once the Validation Suite is enforcing the Ground Truth Contract and the Semantic Drift diagnostic is in place, the question is what prevents failures from reaching production in the first place. Most Semantic Drift Vectors are detectable before a prompt ships if you run the right adversarial test before deployment. That test is structured and repeatable. It targets the failure modes in your Ground Truth Contract that no reference set will surface until production has already seen them. It has a name.
The Constraint