AI-300: Lab: Testing, Rolling Out, and Reverting an Online Model

Introduction

Rehearse an online model release by keeping a known-good deployment available while testing a separate candidate, observing it with mirrored traffic, and moving ordinary requests in controlled stages. Prove rollback by restoring routing to the retained deployment and verifying its responses and capacity. A saved old model file or an accepted routing update alone is insufficient evidence that recovery works.

This is a constrained sandbox design rehearsal for Azure Machine Learning managed online endpoints. It includes expected checks and educational command patterns, with no claimed cloud execution or fabricated output. The scenario moves through direct candidate testing, optional mirroring, a small live allocation, full candidate allocation, and deliberate reversion. Batch routing and Microsoft Foundry model upgrades are outside its scope.

Troubleshooting Methodology and Sandbox Boundary

Assume an illustrative demand-prediction service with an existing deployment named blue and a qualified candidate named green, both under one disposable managed online endpoint. “Live traffic” in this rehearsal means ordinary, deployment-unspecified sandbox requests whose returned predictions are consumed by the test client. It does not mean production customer traffic.

Start after AI-300: Choosing Online and Batch Inference. Before execution, establish:

• An authorized non-production workspace and an endpoint reserved for this exercise, with exact subscription, resource group, workspace, and endpoint recorded.
• A qualified baseline and candidate, compatible request contracts, and recorded model, environment, and scoring-code versions.
• The reviewed official managed-endpoint sample files and their source revision. The safe-rollout guide locates configuration under cli/endpoints/online/managed/sample/ and its accompanying script in cli/.
• CLI v2 tooling, authorization for online-endpoint operations, and a usable network path from the test client. Deployment runtime dependencies must also be accessible.
• Quota and approved capacity for both deployments simultaneously, including the documented upgrade allowance. Do not assume the old deployment can be shrunk away and still provide immediate recovery.
• A maximum test duration, bounded request rate, spending limit, cleanup owner, and prewritten stop/revert criteria. Budget notifications are not assumed to stop resources automatically.

Use synthetic inputs and side-effect-free scoring. Mirroring copies requests, so the test must not accidentally duplicate real writes or external business actions. Do not log endpoint credentials or sensitive payloads. Retain sanitized request identifiers and results instead.

Before each change, record the expected routing state and the observation that would justify the next step. Freeze unrelated model, identity, and environment changes during the exercise so that failures can be attributed to the candidate or routing transition being tested.

Common Issues to Anticipate

Candidate Readiness Is Not Request Acceptance

Separate resource provisioning, model loading, input parsing, inference, dependency access, and output validation. A candidate can exist without satisfying the consumer contract. Define a representative request set with expected output structure and application-relevant checks; include malformed input and a dependency-failure case in an authorized controlled test.

Choose latency, error, and output-quality criteria before viewing results. The criteria depend on the application; this guide supplies no universal threshold. Keep the same input definitions and load assumptions when comparing the baseline and candidate.

Mirrored Predictions Are Not Client Predictions

The official managed-endpoint guidance distinguishes three paths:

As verified on October 15, 2026, mirroring is unsupported for Kubernetes online endpoints, can target only one deployment, and is capped at 50%. A deployment receives either mirrored or live traffic, not both. The documented minimum tooling versions are Machine Learning CLI 2.4.0 and Python SDK 1.0.0; older-tool endpoint updates can lose mirror settings. Verify the actual tools and settings used by every updater.

Mirror observations can expose latency and errors, but they do not show that clients accepted the candidate's outputs. Retain direct output tests and separate live-path checks. Also do not infer that a small sample must contain exactly the configured percentage of candidate calls.

The Old Deployment Must Remain Usable

Rollback depends on the old runtime, its identity and dependencies, its input compatibility, and sufficient capacity for the restored load. Retaining a name in a routing record does not prove these conditions. Avoid concurrent dependency changes that make the old deployment unusable while the candidate is being evaluated.

Before beginning the candidate transition, invoke the baseline and record representative responses and the capacity assumption for full endpoint load. Repeat relevant baseline checks before the deliberate reversion. If the fallback cannot pass its own checks, stop the rollout exercise and repair that condition first.

Troubleshooting Workflow: Execute One Gate at a Time

1. Establish the Baseline and Create an Isolated Candidate

Review the official sample rather than inventing a deployment schema. Adapt it only after checking model references, runtime dependencies, instance choices, and network requirements. Confirm blue handles all ordinary sandbox traffic and that its predictions meet the baseline criteria.

Create green from its reviewed configuration without allocating live traffic to it. In the official CLI example, omitting explicit traffic allocation leaves the new candidate at zero. Inspect the actual endpoint state after creation instead of assuming that a successful create operation preserved every setting. Do not use an all-traffic creation option for this isolated candidate stage.

Acceptance evidence is the resolved baseline/candidate configuration, completed provisioning, and observed live allocation of blue=100 and green=0, with mirroring disabled. These are expected states for the exercise, not output observed while writing this article.

2. Invoke the Candidate Directly

The following Bash pattern assumes the exact sandbox target and CLI workspace/group defaults have already been checked. Set ENDPOINT_NAME to that target. The request path is from the reviewed official example repository; substitute a compatible reviewed request file if using your own candidate.

az ml online-endpoint invoke \  --name "$ENDPOINT_NAME" \  --deployment-name green \  --request-file endpoints/online/model-2/sample-request.json

Run the representative acceptance cases and record output structure, prediction checks, errors, and latency under the intended test conditions. Confirm ordinary calls still return baseline responses. Direct candidate success demonstrates the named deployment's behavior; it does not test traffic splitting.

If initialization, schema handling, or dependency access fails, leave ordinary traffic on blue. Preserve the first useful error and investigate the failing layer. Do not compensate for a failed direct test by moving a small live share to the candidate.

3. Observe Optional Mirrored Traffic

After direct acceptance, configure a modest shadow share within the documented limit. The official guide uses 10% as an example; it is not a universal safety threshold. Generate a bounded set of ordinary requests without a deployment override. Inspect candidate logs and metrics for received work, errors, and latency while confirming that clients continue to receive baseline predictions.

az ml online-endpoint update --name "$ENDPOINT_NAME" --mirror-traffic "green=10"

Record the observed mirror setting, input/load context, and candidate observations. Sparse mirrored work is inconclusive; do not claim statistical confidence from a few requests. If no candidate work appears, first check whether requests were explicitly targeting a deployment, whether traffic was generated, and whether a subsequent tool update changed the mirror setting.

Before proceeding to live allocation, disable mirroring and verify the setting:

az ml online-endpoint update --name "$ENDPOINT_NAME" --mirror-traffic "green=0"

Stop here if mirror evidence breaches the predefined criteria. The baseline remains the live path; diagnose the candidate before attempting another transition.

4. Allocate Live Traffic in Stages

Record the approved pilot allocation and observation window. The documented blue=90 green=10 example is a useful rehearsal state. Apply it only after confirming mirroring is off and both deployments are ready. Continue generating ordinary, deployment-unspecified requests and verify the actual routing state and client-visible outcomes.

The table below is the proposed rehearsal plan, not an automatic Azure release policy. Percentages total 100 for each active live-routing state.

At each stage, compare the actual endpoint configuration with the intended state. If errors, unacceptable outputs, or latency cross the agreed trigger, stop advancement and revert. A completed routing update is control-plane evidence; confirm the request path separately.

5. Revert and Prove Recovery

Keep blue provisioned through the candidate observation and deliberate recovery test. Restore its full live allocation while removing the candidate's live share; ensure no mirror allocation remains. The CLI update pattern is the same documented traffic-setting operation with the original allocation restored:

az ml online-endpoint update --name "$ENDPOINT_NAME" --traffic "blue=100 green=0"

Inspect the saved endpoint state, then send ordinary requests without a deployment override. Confirm baseline response checks, acceptable latency/error behavior at the agreed test load, and continued access to dependencies. Directly invoking blue alone would not prove the endpoint's ordinary routing recovered.

The exercise passes its recovery gate only when those observations support the restored service. Record the change, request observations, and any residual failures. Do not promise that routing reversion undoes external side effects or reverses an incompatible shared-data change; the sandbox constraints deliberately avoid those complications.

Failure Analysis and Resolution Record

Change one candidate variable at a time after preserving sanitized failure evidence. A revised model, runtime, or request contract needs new acceptance observations; do not reuse the earlier candidate decision simply because the deployment name is unchanged.

Best Practices for Cleanup and Cost Protection

Both deployments consume resources while retained, even when one receives no live traffic. Mirroring adds work. Bound the duration and load, inspect failed-run resources, and do not assume a zero traffic allocation stops compute charges.

For this rehearsal, the default final service state is the restored baseline. Preserve its accepted recovery evidence, stop test clients, disable mirroring, and confirm green has zero live allocation before removing the rejected or unneeded candidate. The official guide requires zero live traffic before deleting an individual deployment. Verify deletion completed rather than treating an asynchronous request as proof of removal.

Do not remove the old deployment while it is still the intended fallback. If an authorized later decision retains the candidate instead, first establish the accepted final allocation and recovery disposition, then verify the old deployment has zero live and mirror allocation before deleting it. The prior fallback is no longer immediately available after deletion; record that lifecycle change explicitly.

If the entire disposable endpoint is no longer needed, confirm its exact ownership and dependencies before deleting it; endpoint deletion also removes its underlying deployments. Retain required evidence and account for any separately retained models, environments, logs, or other resources. Never delete a shared workspace or resource group merely because this endpoint exercise has ended.

Certification Relevance

This exercise develops the ability to separate candidate behavior, shadow observations, client-visible release behavior, and actual recovery. For longer-term production model signals and retraining decisions, continue with AI-300: Monitoring Drift, Actual Performance, and Retraining. The sandbox's rollout checks do not replace that operational monitoring loop.

Conclusion

A defensible online rollout keeps the candidate isolated until tested, distinguishes mirroring from client-visible traffic, and advances only on observed evidence. Rollback remains credible when the old deployment is still usable and ordinary requests succeed after routing is restored. Cleanup follows the accepted final state, with allocations removed and costs accounted for before resources are retired.

Authoritative Sources

• Endpoints for inference in production
• Safe rollout for online endpoints

Related Reading

• AI-300: Choosing Online and Batch Inference
• AI-300: Monitoring Drift, Actual Performance, and Retraining