AI-300: Lab: Comparing RAG Retrieval Configurations

Introduction

Test a retrieval change by running baseline and candidate against the same frozen questions and source snapshot, then measuring retrieved evidence and generated answers separately. Change one retrieval factor at a time, keep generation settings and context-selection rules fixed, and inspect paired failures. Better ranking is useful only if the evidence needed by the application survives into context and the resulting answers meet the acceptance criteria.

This is an isolated sandbox design rehearsal for comparing retrieval-augmented generation (RAG) configurations. The operations and example values below are proposed, not executed: no index, model call, evaluation result or winning configuration is claimed. “Offline A/B” means a paired comparison on a test set here; it does not mean randomized live traffic or demonstrated production impact.

Scenario and Intended Outcome

An internal support assistant searches synthetic device manuals. Some questions use exact model/error identifiers; others paraphrase symptoms. The team wants to test whether a chunking change, greater vector candidate depth, or a hybrid strategy improves usable evidence without increasing unsupported advice or unacceptable latency.

Produce a comparison record that answers three separate questions:

• Did the candidate retrieve more of the required evidence at a useful rank?
• Did the generated answers improve, remain equivalent, or regress under the agreed review criteria?
• Were any gains worth the measured latency, resource use and operational complexity?

Use the gate principles from AI-300: Choosing Quality, Safety, and Custom Evaluation Gates. The lab's deliverable is a reviewable decision, which may be “retain the baseline” or “inconclusive.” It is not a requirement to find a winning candidate.

Prerequisites and Constraints

Use an authorized nonproduction Azure AI Search environment, an accessible embedding model, a generation deployment, and permission to operate only the named sandbox resources. Azure AI Search does not host the embedding model; use the same embedding model for index and query vectors. Verify the chosen input sizes and API/SDK support before implementing the example configuration.

Prepare approved synthetic or non-sensitive source material, stable source identifiers, a versioned application runner, and evaluation inputs with reviewed evidence labels. Set a spending limit, cap indexing and model/evaluator calls, and identify who will remove temporary resources. Do not direct production clients or indexers to a candidate index.

Keep the scope deliberately narrow: no embedding adaptation, fine-tuning, production rollout or automatic promotion. Use stable supported query functionality for the core comparison. Preview thresholds are discussed as a boundary rather than enabled in this rehearsal.

Prepare a Frozen Experiment Contract

1. Freeze source evidence and query groups

Record the raw-document revision or digest before constructing indexes. Label expected evidence using source-document passages, not generated chunk IDs, because a chunking variant will change those IDs and boundaries. Preserve a mapping from each indexed chunk to its originating passage range.

The following original design cases illustrate labels to create. The device names and instructions are synthetic exercise data, not real repair guidance:

Expand these cases into a broader development set with enough source text for at least ten eligible indexed chunks in each compared index. The table alone is a workflow illustration, not a representative benchmark. Reserve a separate held-out query set with reviewed evidence labels before tuning; do not inspect its candidate outputs while selecting a variant.

Freeze query text, eligibility rules, expected evidence and answer criteria. Keep reference answers and labels in the evaluator input, separate from the context sent to the generation model. Otherwise the experiment may accidentally supply the answer rather than test retrieval.

2. Record what must remain constant

Retain the embedding model/configuration, query vectorization, text-field configuration, vector algorithm, access rules, generation prompt/deployment/settings, context-selection algorithm, evaluator versions and timing method. Store the exact candidate definition beside each run identity.

Use a consistent context budget and selection rule for every configuration. For example, select the first three distinct retrieved chunks, within the same total token budget, in rank order. The actual context will change with retrieval, but the rule for constructing it must not change unnoticed.

When testing chunking, freeze the raw documents while allowing their indexed representation to change. Build a separate candidate index, recompute embeddings for the candidate chunks using the same model, and record its chunk/source mapping. Do not silently overwrite the baseline index or reuse vectors belonging to different text.

Run Isolated Retrieval Comparisons

3. Define baseline and one-factor candidates

The following are illustrative experiment values, not Azure defaults or performance recommendations. Confirm that every eligible corpus has enough chunks and that both window sizes fit the selected embedding model. Use the same tokenizer for both chunking policies:

Compare C against B, K against B, and H against B. Do not combine the three changes and attribute the result to one of them. A chunking change necessarily changes the represented text and available chunks; treat that as the intervention and keep the surrounding pipeline fixed.

Vector k controls nearest-neighbor candidates. Hybrid top controls the response size; neither automatically defines how many passages enter generation. Increasing candidate depth may produce no benefit to the selected context, which is a valid finding.

Semantic reranking is excluded from this first comparison to avoid changing another layer. If investigated later, define a new experiment; the current query guide recommends vector k=50 when using semantic ranking, which accepts up to 50 matches. Do not simply turn it on while retaining an inadequately defined input pool.

4. Capture retrieval before generating answers

For each development query and configuration, execute the recorded retrieval request against its named sandbox index. Save query/configuration identity, ordered chunk IDs, source-passage mapping, query mode, stage-specific scores and retrieval timing. Check that requests used the intended source revision and eligibility rules.

Measure evidence independently of the search score. For this rehearsal, define evidence coverage at three as the number of required labeled source passages represented by the first three selected chunks, divided by the number of required passages. Count repeated coverage of one passage once. Also record the rank of the first required passage and whether all passages for multi-evidence questions survive context selection.

That coverage definition is an application-owned measure, not a claimed built-in evaluator metric. Count a passage as represented only when its answer-bearing content, including necessary conditions, is present in the selected context; a partial text overlap is insufficient. For no-coverage queries, mark evidence coverage as not applicable rather than dividing by zero, then assess insufficient-evidence behavior separately. Inspect examples where a relevant chunk is present but outside the context-selection boundary.

Azure AI Search transforms cosine-based pure-vector search scores; they are not raw cosine similarity. Hybrid results use Reciprocal Rank Fusion (RRF), a different scoring scale. Preserve scores for diagnosis within their query context, but do not declare H better than B because its numerical score changed.

5. Generate and score answers separately

Build each answer request from the recorded context using the unchanged prompt and supported generation settings. Save the actual context, generated response, cited source IDs, timing, errors and available usage evidence. Do not regenerate from a different retrieval result later and label it the same run.

Evaluate whether answers are supported by their actual supplied context, address the query, cite appropriate evidence and satisfy the chosen safety/application rules. Foundry's catalog distinguishes groundedness and relevance from fluency/coherence and risk signals. Use the previously agreed criteria and inspect per-record failures rather than relying only on a quality average.

If evaluating saved responses, score those stored outputs. If invoking a Foundry model/agent target, the service generates new outputs and ignores a stored row response; map the evaluated output and supporting context accordingly. These paths answer different questions, so select and record one deliberately.

Measure retrieval time, generation time and end-to-end time separately. Use repeated paired runs where variation matters, interleave baseline/candidate work to reduce run-order bias, and record cold/warm conditions and errors. Token counts are usage evidence; do not present them as complete billed cost.

Acceptance and Held-Out Review

6. Decide from paired evidence

Choose tolerances before examining candidate results. For this rehearsal, require complete run evidence, no unresolved critical safety or unsupported-action failures, coverage of required multi-passage evidence, acceptable answer quality for each query group, and the agreed latency/resource budget. Numerical floors and regression allowances must be chosen for the application's risk and reviewed examples; none is a universal Azure threshold.

Use this interpretation table when writing the decision:

Leave measured-result fields empty until execution. The table states possible interpretations and is not a report of completed tests.

Select a candidate using development results, freeze its configuration, and assess it against B on the untouched held-out queries. If it fails, record that outcome rather than tuning against the same set and continuing to call it held out. Further selection requires a newly reserved assessment set. A small test set cannot establish reliable production-wide effect sizes.

Failure Analysis and Preview Boundaries

When expected evidence is missing, inspect source ingestion and chunk/source mapping first, then query fields and candidate retrieval. If evidence reaches the returned set but not the prompt, inspect context selection. If correct evidence reaches the prompt and the answer still fails, investigate generation and evaluator judgment rather than automatically increasing k again.

Nearest-neighbor search can return weak matches even when the topic is absent. The current threshold parameter is preview and filters vector results before fusion. A pure-vector threshold is not a universal cutoff for final hybrid RRF scores. Keep threshold experimentation separate, version-qualified and unsupported by any claimed value from this rehearsal.

Also verify evaluator feature status. Preview-marked built-ins carry documented limitations, including no SLA and a recommendation against production workloads. A judge flag or a fluent response is evidence to inspect, not a correctness guarantee.

Cleanup, Cost Protection and Lab Value

Export the permitted experiment configuration, source digests, chunk mappings and actual review evidence before teardown. Stop further indexing and evaluation submissions, remove the named temporary candidate indexes and exercise artifacts, and check for retained model deployments or other resources that continue to incur cost. Remove shared resources only when their owner confirms they belong solely to this exercise.

Confirm the baseline/production configuration was never redirected and document which approved resources remain. Keep sensitive inputs, credentials and unrestricted traces out of the comparison report. Record cleanup completion and any owner-authorized retention explicitly; no cleanup is claimed to have occurred here.

The useful lab skill is separating retrieval evidence from answer evidence under a reproducible comparison. A negative or inconclusive result is still valuable when it identifies the failing layer and preserves a fair next experiment.

Conclusion

Freeze raw evidence, isolate retrieval changes and retain the context actually used for every answer. Compare both evidence coverage and answer behavior, then test a selected configuration on held-out queries under the full acceptance policy. That makes ranking gains, answer gains and operational tradeoffs distinguishable without claiming an unperformed deployment or live A/B result.

Authoritative Sources

• Evaluation dataset schema in Microsoft Foundry
• Built-in evaluators reference
• Relevance in vector search
• Create a vector query in Azure AI Search
• Generate embeddings for search queries and documents

Related Reading

• AI-300: Choosing Quality, Safety, and Custom Evaluation Gates
• AI-300: Choosing Supervised, Preference, and Reinforcement Fine-Tuning