Your office manager has been using the same AI tool for months. She's got a prompt she trusts: paste in a client call transcript, get five clean bullet points, drop them into the CRM note. It works. She stops thinking about it.
Then one Tuesday it gives her three paragraphs instead of five bullets. She re-runs it. Three paragraphs again. She assumes she did something wrong, tweaks the prompt, gets something close enough, and moves on. Nobody flags it. The process continues — slightly broken, silently degraded — until someone eventually notices the CRM notes have been inconsistent for weeks.
That is version drift. It's different from the run-to-run variation where the same model gives you slightly different answers each time you ask, and it's different from the context window issue where the AI "forgets" things from earlier in a conversation. This is something else: the vendor swapped the engine underneath a product you thought was stable, and they did it without asking you.
The Product Name Stays. The Engine Doesn't.
When you subscribe to an AI writing tool, a meeting summariser, or a customer support assistant, you're not buying a frozen piece of software the way you'd buy a calculator. You're renting access to a model — and that model gets replaced, refined, and retrained constantly.
The product name stays the same. The logo stays the same. Your invoice looks identical. But underneath, the vendor has upgraded from one version of the underlying language model to a newer one, or switched providers entirely, or fine-tuned their existing model on new data. They usually announce it somewhere — a blog post, a changelog page, a banner in the dashboard you clicked past — but the update goes live whether you read it or not.
This isn't a scandal. It's just how the industry works. Vendors are under competitive pressure to keep their tools feeling current, and "we upgraded the model" sounds like good news. Often it is, in the aggregate. The new model scores better on the benchmarks vendors care about: reasoning, factual accuracy, language fluency.
The problem is that "better on benchmarks" does not mean "better at your specific task."
A model that gets stronger at logical reasoning can, at the same time, get worse at following a rigid output format. A model trained to be more conversational and helpful will start adding a friendly preamble — "Great question! Here's what I found:" — that breaks a copy-paste workflow into an invoice template. A model improved for nuance might stop returning crisp yes/no categorizations that a routing process depends on. Better at the thing the vendor measures. Worse at the thing you needed.
The Symptoms You'll Recognize
Version drift doesn't usually announce itself with an error message. It shows up as a vague sense that something's off. Some patterns to watch for:
The format changed. You used to get five bullets. Now you get paragraphs. Or the headers are gone. Or the output is in a different order than the template expects. Your downstream process — the one that pastes directly into a doc, a form, or a message — suddenly needs human editing every time.
A preamble appeared. The tool started adding a conversational opener before the actual content. Fine if you're reading it for personal use; breaks things if the output feeds directly into something else.
A category it used to get right is now wrong. You trained your team to trust the tool's ability to classify incoming inquiries into three buckets. It's been quietly miscategorizing one of them for six weeks. The emails still go somewhere — just not the right place.
The output got longer. Something that used to return a tight one-paragraph summary now returns 400 words. Nobody complained because longer feels more thorough. But the person reading summaries just started skimming them instead of acting on them.
The tone shifted. The tool was appropriately direct. Now it hedges. Or it stopped hedging and got breezy. Either way, the voice no longer sounds like yours.
None of these are catastrophic on their own. Combined and unnoticed, they quietly erode the value you thought you were getting.
The Second Failure Mode: Things Get Retired
Version drift is the slow version. There's also the sudden version.
Vendors discontinue models and features with varying amounts of notice. A specific model version you built a process on — maybe you accessed it via an API, or chose it explicitly in the tool's settings — gets end-of-lifed. Your workflow breaks in a way that is at least visible. But if you didn't document what you built or why, the person fixing it is starting from scratch.
The risk of depending too heavily on a single vendor's specific implementation is worth understanding in depth. If you've already read about vendor lock-in and what to do if your AI tool disappears, you'll recognize the pattern: the platform keeps running, but the version of it you relied on is gone, and nobody on your team quite remembers how the process was set up.
Why This Is Different From Normal Software Updates
You're probably used to software updates. Excel updates. QuickBooks updates. Your browser updates. But a formula in a spreadsheet does what it did last year. =SUM(A1:A10) returns the same result it always has. The logic is fixed.
An AI model doesn't work that way. It's a statistical system — it generates outputs based on patterns learned during training. When the training changes, the patterns change. There's no "the formula" that stays constant. The whole point of updating a model is to change how it behaves. And "how it behaves" includes the things you were depending on.
This is also distinct from the variation you see within a single session. As explained in the piece on why AI gives different answers to the same question, there's inherent randomness baked into how language models generate text — which is a separate issue from what happens when the model itself is replaced. Run-to-run variation is noise around a stable center. Version drift is the center moving.
The Practical Response (It's Unglamorous, and That's the Point)
None of what follows is technically complicated. Most of it takes an afternoon to set up and about thirty minutes a quarter to maintain. That's the point.
Keep your important prompts in a file you control.
If you built a prompt that reliably produces the output you need, that prompt is an asset. Don't let it live only inside the tool's interface, embedded in a workflow someone set up eighteen months ago. Copy it into a shared document — a Google Doc, a Notion page, a plain text file, anything your team can access. Include a note about what it's for and roughly when it was last confirmed to work.
This costs nothing and takes five minutes per prompt. The alternative is trying to reconstruct from memory something that took you three hours of iteration to get right.
Keep three or four real examples with expected outputs.
Pick the most important task your team uses AI for. Take three or four real inputs you've actually run, and save them alongside the outputs you considered acceptable. When something feels off, re-run those examples. If the output has changed noticeably, you have a documented baseline to work from — and you've confirmed the problem is the model, not the operator.
This is the AI equivalent of a calibration check. Not sophisticated. Very useful.
Put a date in the diary each quarter.
Set a recurring calendar reminder — quarterly works for most businesses — to spot-check any AI process running unattended. Ask: Is it still producing what we expect? Has anyone flagged anything unusual? Run a test example or two. This takes fifteen minutes if everything is fine, and catches problems before they become six-week-long data quality issues.
Anything that runs automatically, touches customer-facing content, or feeds into a business decision deserves this check. Anything a human reviews every time is lower risk — their eyes catch the drift. It's the unattended, automated, or semi-automated workflows that go quietly wrong.
Make one person responsible for reading release notes.
Someone at your company should have a tab open to the changelog or release notes for each AI tool you rely on. Not every update matters. But when a vendor announces a model upgrade or a feature change, you want to know before it happens — not three weeks after the symptoms showed up.
This doesn't require an expert. It requires someone willing to scan a short document once a month and flag anything that mentions model versions, output changes, or deprecated features. In a smaller team, this might be one person covering all tools. Rotate it if you have to. Just don't assume it's happening if nobody owns it.
The Right Mental Model
Think of an AI tool the way you'd think of a supplier who occasionally changes their recipe without telling you.
Your coffee supplier changes the blend. Your paper vendor switches mills. The product looks identical, the price is the same, and most customers don't notice. But if your process depends on a specific property of that product — a specific grind, a specific thickness — you need to catch the change and adjust before it causes problems downstream.
You don't treat a supplier relationship as permanent and static. You verify quality. You stay in contact. You keep a sample of what good looks like so you can compare.
A spreadsheet formula doesn't need that treatment. An AI tool does. The sooner that becomes a habit in how your team thinks about these tools, the fewer Tuesday-morning surprises you'll deal with.
If you have AI workflows running in your business and you're not sure how stable they are or who's watching them, that's a reasonable thing to get a second pair of eyes on. Book a strategy call and we can talk through what you have and where the exposure is.