AI Model Deprecation: How to Migrate Without Breaking Production

Why model migrations happen

Model providers retire or replace versions for many reasons: product strategy, infrastructure changes, security improvements, cost, new capabilities, or the end of a preview period. Provider documentation from organisations such as Anthropic and Google describes model lifecycle states, replacement recommendations and shutdown dates. A retirement can affect an application even when the model name appears only in one configuration variable. Prompts, parsing rules, evaluation assumptions, safety checks and support instructions may all have evolved around the old model’s behaviour. Treat the notice as the start of a migration project. Confirm the actual shutdown date, affected endpoint, replacement options, rate limits and any stated compatibility differences from the provider’s current official documentation.

Inventory where the model is used

Start by finding every reference to the model across production, staging, scheduled jobs, background workers, evaluation pipelines and customer-specific configurations. Record the owner, purpose, traffic level, data classification, failure impact, request volume and downstream systems. Include indirect use through orchestration frameworks or vendor products, not only direct API calls. An internal model registry or ordinary configuration repository can help, provided it is kept current. Note whether a workflow relies on long context, image or audio input, structured responses, tool calling, JSON modes or a particular refusal pattern. Identify fallback models and whether they have actually been tested. This inventory makes it possible to prioritise migrations by business risk instead of discovering a hidden dependency after the endpoint begins returning errors.

Build the evaluation set from real tasks

A replacement should be evaluated against the job the product promises, not a handful of prompts that made the old model look good. Use representative, privacy-safe examples from common requests, edge cases, ambiguous inputs and known failure scenarios. Define what counts as a correct or useful outcome before scoring outputs. Some tasks can be checked deterministically, such as valid JSON, required fields, citations that resolve, or tool calls that obey a schema. Other tasks require a trained human reviewer using a clear rubric. Keep examples that previously caused harm or costly rework. Remove sensitive information where possible and ensure you have the rights and controls required to use the test set. Re-run the same cases after prompts or model settings change so comparisons stay meaningful.

Compare outcomes, not just the model’s reputation

Record task success, factual error rates, refusal behaviour, tool-call accuracy, human edit time, latency, reliability and cost per accepted outcome. A model that produces longer, more polished answers may still be worse if the product needs structured decisions or concise support. A model that scores well on a public benchmark may struggle with your own language mix, domain terminology or workflow state. Analyse results by task segment and failure severity rather than relying only on an overall average. When a model migration changes quality, the difference may show up in retrieval use, tool selection or format compliance instead of factual knowledge alone. Write down the decision threshold and what evidence would cause you to reject the candidate or restrict its use.

Test the whole integration contract

Model behaviour is only one part of compatibility. Verify request fields, response shape, token limits, tool schemas, streaming events, safety settings, timeout handling, rate limits and error codes. Check how the application handles missing fields, partial JSON, empty responses, refusals and tool calls that request unavailable actions. A migration can silently break when a parser expects a format that the new model no longer produces consistently. Run tests with the real orchestration layer and the same configuration used in production. Keep secrets in server-side configuration and avoid putting provider credentials into prompts or client code. Confirm monitoring and billing telemetry still record the right model and usage. This integration test should be separate from judging the semantic quality of the answers.

Check latency and cost under representative load

Compare performance with realistic input lengths, output lengths, concurrency and traffic patterns. An average response time can hide a long tail that affects customers during peak use. Measure the full request path, including retrieval, moderation, network calls, tool execution and application processing. Track time to first token where streaming is used, total completion time, timeout rate and retries. Calculate cost per successful task, not only cost per million tokens or per API call, because a weaker model may trigger extra retries, longer answers or more human review. Include any change in context-window usage, batch processing, caching and rate-limit constraints. Run a load test within provider rules and compare results using the same workload and measurement method where possible.

Use a feature flag and canary rollout

Keep the old and new model available long enough to compare safely when provider terms and technical constraints allow it. Route a small, representative share of traffic to the new configuration and tag every request so outcome and cost can be inspected. A canary should include the real distribution of requests, not only internal prompts written by the migration team. Decide in advance what constitutes a stop signal: a material rise in severe errors, tool misuse, timeout rate, user complaints, human edits or cost. Increase traffic in stages only when the evidence meets the threshold. Use a configuration or feature flag rather than duplicating application logic. The goal is to contain an unexpected behaviour change while keeping a safe path for users.

Make rollback realistic and reversible

A rollback plan is useful only if it has been tested. Confirm the prior model endpoint remains available and permitted for the required time, credentials still work, application configuration can be restored quickly, and data or prompts have not changed incompatibly. If the former model will be retired imminently, designate a second fallback or a safe reduced-capability mode. For a high-risk operation, stopping the feature may be safer than automatically switching to an untested alternative. Document which team can trigger rollback, what conditions require it, and how customer support will communicate the effect. Check that rollback does not replay financial transactions or other side effects. Store enough logs to diagnose the incident without exposing private prompts or credentials unnecessarily.

Update monitoring, documentation and ownership

When a new model passes validation, update the model registry, runbooks, architecture records, evaluation baselines and vendor-dependency list. Tell support and operations what changes in observed behaviour or known limitations. Keep an eye on quality after launch because live traffic evolves beyond the test set. Monitor changes in input distribution, user feedback, tool errors, cost and the frequency of human intervention. Assign a person responsible for reviewing future deprecation notices and maintaining a model lifecycle calendar. This small ownership step prevents a familiar failure: the team migrates successfully once, then nobody notices that the replacement itself is approaching end of support. Save both positive and negative evaluation results so later migrations can build on evidence instead of beginning from memory.

A practical migration sequence

A compact sequence is: discover dependencies, confirm the official retirement timeline, define test criteria, run a baseline, compare replacement candidates, validate API integration, load-test, canary, review live signals and either expand or roll back. The sequence should be tailored to risk. A low-impact drafting feature may require limited safeguards, while an agent authorised to update customer records needs stronger approval boundaries and incident readiness. Preserve the option to limit feature scope if no candidate performs adequately. Avoid promising users unchanged behaviour unless you can show evidence for the tasks they care about. Record unresolved differences explicitly. A model upgrade may create new capabilities, but a controlled migration makes sure those capabilities are evaluated rather than assumed to be beneficial.