AI-300: Preparing Evaluation Datasets and Field Mappings

Prepare evaluation records by first deciding whether Microsoft Foundry will score stored responses or invoke a model or agent target to generate new responses. Choose an interaction format, identify the fields required by the selected evaluators, and explicitly map any nonstandard dataset columns or generated target outputs to those inputs. Then inspect a small sample for both structural correctness and semantic alignment before launching a larger run.

The critical distinction is the response's origin. A stored-output evaluation scores the response in the record; a target evaluation generates a fresh response and ignores a response already present in that row. A valid file can therefore evaluate the wrong thing if the run mode or mapping is wrong. This procedure uses current Microsoft Foundry dataset documentation verified on October 15, 2026; it does not combine current mapping expressions with an older evaluation SDK's configuration syntax.

What an Evaluation Dataset Must Represent

An evaluation dataset is a collection of cases that supplies the interaction and any supporting information needed for scoring. JavaScript Object Notation Lines (JSONL) represents each test case as one JSON object on its own line. Parsing those objects successfully establishes only a structural starting point: the record still has to describe the intended interaction.

Separate three responsibilities in the preparation record. The target contract defines what an invoked application receives. The evaluator contract defines what evidence is needed to score its result. The dataset supplies cases and references that meet those contracts. A field intended for an evaluator should not silently become an instruction or answer hint sent to the target.

This preparation matters because a missing response, misplaced reference answer or unrelated context can make the evaluation uninterpretable. Improving the apparent score is not the goal of data mapping; making each evaluator receive the intended evidence is.

Step 1: Fix the Run Mode and Interaction Format

For existing interactions, Foundry supports a messages array or separate query and response fields. Messages can preserve history and tool interactions; separate string fields are useful for straightforward single-turn cases. Use one interaction representation per case rather than supplying competing accounts of the same conversation.

For a model or agent target, supply the input through messages or query; Foundry generates the response. In the messages form, the documented target-input pattern ends with a user message, whereas a stored interaction includes the assistant response being evaluated. Preserve the appropriate history instead of flattening a multi-turn case into a question that no longer has its original meaning.

Record the chosen mode with the dataset version and run configuration. As a recommended naming practice, distinguish stored-response runs from target runs in internal records. Identical questions do not make those runs equivalent: one evaluates captured behavior, while the other exercises the currently invoked target.

Step 2: Inventory Required Evaluator Fields

Start with the selected evaluators' current input requirements. There is no universal rule that every case must contain every possible supporting field.

For example, the dataset documentation identifies reference answers for textual comparison, supporting context for groundedness with string inputs, and tool definitions for evaluators requiring tool schemas. These requirements are conditional. A blank string is not a meaningful replacement for missing supporting evidence.

Use context and ground truth for their different purposes. Context provides source material that may support an answer; ground truth supplies a reference answer. Copying one into the other without considering its meaning can change what the evaluation measures. Detailed metric selection and release criteria belong in AI-300: Choosing Quality, Safety, and Custom Evaluation Gates.

Step 3: Author a Small Original JSONL Sample

The following fictional library examples illustrate stored responses with nonstandard column names. They are original teaching data, not claims about an actual library policy or outputs from an executed model. Each line is a complete JSON object; the names deliberately require a mapping.

{"question":"How long is a standard loan?","answer":"A standard loan lasts 14 days.","passage":"In this fictional library, a standard loan lasts 14 days.","reference":"The loan period is 14 days."} {"question":"Are renewals allowed?","answer":"The supplied policy does not state whether renewals are allowed.","passage":"In this fictional library, a standard loan lasts 14 days.","reference":"Renewal permission is not specified in the supplied policy."}

The second case deliberately tests a gap in the supplied information. Its context and reference are aligned with an answer that acknowledges that gap. This is a recommended dataset-design pattern for avoiding a sample that tests only easy information lookup, not a claim that either record has received a passing evaluator result.

For a real dataset, retain provenance and case identifiers through an appropriate dataset or sidecar record. Record whether cases were authored, curated or collected under an approved process. Review sensitive content before storing or sharing it, and preserve the information necessary to interpret each interaction.

Step 4: Map Dataset Fields and Generated Outputs Explicitly

For the stored-output example, the intended mapping is:

This table defines the data contract; it is not a complete SDK request or executable configuration. Configure the corresponding mapping in the chosen current Foundry workflow. Compatible evaluators using standard column names can omit data_mapping; nonstandard names such as question require an explicit mapping. Do not assume an evaluator will infer semantics from a familiar-looking column name.

For a target run, revisit the response binding. The stored answer is no longer the output under evaluation simply because it remains in the file. Map the evaluator to the response generated by the target. The current dataset documentation specifically illustrates {{sample.output_text}} for an evaluator such as Coherence that needs generated text, and {{sample.output_items}} when a structured-output evaluator needs generated items. These expressions identify different output representations and are not interchangeable.

Keep input mappings and output mappings separate during review. For example, the question should come from the case, while the scored response in a target run should come from its generated output. Supporting evidence must correspond to the case and response being assessed. If an application retrieves evidence dynamically, determine how the actual relevant evidence reaches the evaluator; an unrelated static passage is not a sound substitute.

Do not paste these expressions into a configuration from another generation of Foundry tooling and assume compatibility. This article intentionally defines the record and mapping intent without fabricating an API call or mixing current expressions with legacy SDK examples.

Step 5: Preflight Structure and Meaning

Before a paid or large evaluation run, inspect the actual file and configuration together. Recommended checks are:

1. Parse each JSONL line independently and confirm each intended case has the expected fields and value types.
2. Resolve the selected evaluator's required inputs against the record and mappings. Identify missing, null or empty values instead of quietly replacing them with fabricated content.
3. Check a small number of complete cases manually: question, response origin, context, reference and intended evaluator inputs should tell one consistent story.
4. Confirm whether a target will be invoked. For target evaluation, check the generated-output mapping and make sure a stored response is not being mistaken for the evaluated result.
5. Inspect conversation ordering and message roles where that format is used. Preserve the history needed to interpret the final input or stored answer.
6. Perform a small controlled evaluation in the selected supported workflow, then inspect case-level errors and the fields actually consumed before expanding the run.

The last step is a recommended verification procedure, not execution evidence for this article. Before expanding, confirm the cost and access requirements for the selected target and evaluators, and bound the sample accordingly.

Treat a missing-field error as a preparation failure rather than a poor model score. Likewise, a mapping that routes a reference answer into response can produce a superficially plausible run while assessing the wrong text. Correct the binding and reassess the affected cases; preserving the earlier numeric output does not repair its meaning.

Step 6: Preserve Test-Set Integrity and the Handoff

Freeze the reviewed records and mapping configuration used for a comparison. Recommended evidence includes dataset identity, input mode, selected evaluator requirements, response source and the target/configuration identity when invoked. Preserve case ordering or stable identifiers where needed to trace individual results.

Use a clear process for correcting reference answers and context. If a reference was wrong, revise it deliberately and identify which comparisons are affected. Avoid changing test records to favor a particular candidate while retaining the impression that earlier and later scores are directly comparable.

Keep evaluation references separate from material used to tune prompts or train models when an independent assessment is intended. A technically valid mapping cannot reveal that a supposed held-out case was repeatedly used to optimize the candidate. Dataset preparation should therefore record usage history as well as file structure.

The resulting handoff lets an evaluation owner answer four questions: what input was supplied, which response was scored, what supporting evidence was used and how those fields were bound. Choosing the actual quality/safety measures and their acceptance thresholds is a separate task.

Common Learning and Operational Mistakes

• Treating parseable JSONL as proof that every evaluator has its required inputs.
• Assuming stored responses are scored even when the run invokes a target.
• Sending ground-truth answers to the target as though they were ordinary user inputs.
• Binding generated structured items to an evaluator expecting text without checking its contract.
• Supplying the same context for cases whose actual supporting evidence differs.
• Treating missing data or execution errors as model-quality measurements.
• Changing records or mappings between candidates without preserving the difference.

Conclusion

Prepare the dataset and run mode as one contract. Identify the response origin, supply only the supporting fields required by the selected evaluators, map them deliberately and inspect a bounded sample before scaling. Preserve the reviewed records and bindings so later scores can be traced to the interaction actually evaluated.

Authoritative Sources

• Evaluation dataset schema in Microsoft Foundry

Related Reading

• AI-300: Versioning Prompts and Comparing Variants with Git
• AI-300: Choosing Quality, Safety, and Custom Evaluation Gates