AI-300: Qualifying Synthetic Fine-Tuning Data

Turn synthetic examples into training data through a qualification process: verify source permission and privacy, define the behavior and generation constraints, inspect a small pilot, review and filter the output, keep related cases within one data partition, and freeze an independent evaluation holdout. Retain the source and generation provenance behind accepted examples so a reviewer can explain what they teach and why they were retained. A generated file is a candidate dataset, not evidence of factual correctness, anonymity or improved model behavior.

In Microsoft Foundry, synthetic data generation for fine-tuning is preview. Verify current access, supported region, generator/reference requirements and compatibility with the chosen training method before planning a run. This guide uses official documentation verified on October 15, 2026 and describes a proposed procedure; it does not claim that synthetic generation or training was executed. Method selection precedes this work, and deployment qualification follows it.

What Synthetic Data Can and Cannot Establish

Synthetic data is produced rather than collected as a direct record of real application behavior. It can help expand examples when useful real data is limited, but fluent variations may repeat the same narrow pattern or introduce unsupported claims. Dataset size alone does not establish representative coverage.

Foundry's documented preview includes Simple Q&A generation from a domain document and Tool use generation from an application programming interface (API) specification in OpenAPI format. The former supports short- or long-answer choices; the latter simulates tool-use conversations. The output is JavaScript Object Notation Lines (JSONL) compatible with Foundry fine-tuning workflows and can be downloaded for post-processing.

Compatibility of a generated file with a workflow is only a format claim. The intended examples must still match the selected training signal. A generated response is not automatically a reviewed desired demonstration, a preferred response over an alternative or a dependable grading signal. Carry the supported method and evidence decision from AI-300: Choosing Supervised, Preference, and Reinforcement Fine-Tuning into qualification rather than changing methods implicitly because one file is convenient.

Step 1: Establish Source Permission and Privacy

Before uploading a reference file, record its owner, origin, revision, permitted uses and intended consumers. As a recommended governance check, confirm permission to use it for generation and downstream training, rather than assuming access to read a document establishes those permissions. Resolve restrictions through the organization's responsible process; this article does not assert a universal license or authorization rule.

Remove secrets, credentials, inappropriate personal information and other sensitive material before submitting the reference. Microsoft's generation guidance explicitly advises excluding personally identifiable information and sensitive data. An API description can also contain example tokens, internal addresses or customer identifiers, so inspect examples and descriptions as well as the main prose.

Synthetic transformation is not a privacy guarantee. Renaming a person or paraphrasing a sensitive incident may preserve identifying combinations or the underlying confidential facts. Review generated outputs again for inappropriate reproduction or disclosure; do not use the word “synthetic” as evidence that this review is unnecessary.

Recommended provenance should distinguish deliberately fictional examples from transformed source cases. Keep enough controlled lineage to investigate a defect without spreading the sensitive original into every training record. Decide retention and access for those records explicitly.

Step 2: Check Preview Eligibility and Define Generation Constraints

Check the current project/access prerequisites, supported generation region and usable generator model. Generation availability is separate from the selected model's fine-tuning support and eventual serving eligibility. Preview functionality, formats and limits may change; do not infer production suitability merely from being able to start the wizard.

The documented preview uses one reference file, with document formats for Simple Q&A and an appropriate JSON OpenAPI specification for Tool use. Inspect the current supported versions and file limits before preparing the artifact. Do not assume an arbitrary collection of files or an unrelated API format will be accepted.

Define the target behavior in a generation brief. Recommended constraints include:

• Which source facts and business rules an answer or tool interaction may rely on.
• The expected response structure, level of detail and handling of missing information.
• The task groups, difficult cases and error conditions that need coverage.
• Which sensitive content, unsupported claims and undesirable actions must be excluded.
• The source identifiers or groups needed to track related examples.

These are design requirements to express through the supported reference/configuration surfaces and review process, not invented wizard fields. For tool-use generation, put accurate function, parameter and business-policy descriptions in the reviewed API specification. Do not assume a syntactically valid specification describes correct application behavior.

The generation documentation notes that selecting an undeployed generator model can create a deployment automatically. Check permissions, resource effects and applicable costs before submitting a pilot. No deployment or job is created by the procedure in this article.

Step 3: Protect Independent Evaluation Before Expanding Data

Reserve the final evaluation cases before using generation and filtering choices to optimize the training set. Keep those cases out of reference files, generation examples, reviewer tuning prompts and other material that can influence the candidate dataset. If a final test case repeatedly guides data construction, it no longer provides the same independent evidence.

Partition related cases as groups. Exact text duplicates are only one leakage route: paraphrases of one incident, alternative conversations from one scenario and reformatted copies of the same answer can cross a random row-level split. Identify the relevant grouping unit for the claimed generalization, such as source incident, customer case, scenario template or document family.

For an illustrative maintenance-handover dataset, several rewrites of one underlying incident should stay in the same partition. A holdout containing a polished paraphrase of that incident would mostly test familiarity with its content. Independently reviewed, unseen incidents provide stronger evidence about transfer to new cases. The appropriate grouping depends on what the evaluation is intended to establish; document that choice.

The preview offers an automatic train/validation split. Do not treat that convenience as proof of group separation or as an independent final test set. Inspect related-case overlap yourself and maintain the final holdout under a separate controlled process. Validation used during development and a final assessment have different roles.

Step 4: Generate a Bounded Pilot and Review the Output

Microsoft recommends starting with a smaller sample and inspecting it before scaling. Plan a pilot around useful coverage rather than the largest permitted row count. Retain the generator/model selection, source revision, supported parameters and job identity if an actual run is later performed.

Review both mechanical validity and content quality. JSONL has one JSON object per line, but parseable records can still teach the wrong answer. Inspect role ordering, required fields, tool-call consistency and the exact format accepted by the selected method. Avoid injecting internal provenance columns into a training schema without checking support; maintain a sidecar manifest where necessary.

For human review, define a rubric and inspect samples from every relevant source/task group. Include difficult cases, unusual or very long records, tool failures and cases flagged by automated checks. Use risk-based inspection for consequential behaviors, and resolve reviewer disagreements rather than averaging incompatible judgments.

A model-based filter can help prioritize review, but a generator approving its own outputs is not independent validation. Sampling also cannot establish that every uninspected record is correct. Record which population was checked, how samples were chosen, defect categories and the limits of the review. Recurring defects should trigger a broader source/constraint correction and renewed review, not just deletion of the visible examples.

Step 5: Deduplicate and Check Group Leakage

Perform exact duplicate checks and inspect near-duplicate families. Recommended comparisons should consider the input, intended output and underlying scenario, not only full-row text. Removing a changed timestamp can reveal identical cases; different wording can still encode the same incident.

Review conflicts as well as repetitions. Two examples may give incompatible expected responses to equivalent inputs. Resolve the source ambiguity or intended policy before accepting either as supervision. Repetition of a mistaken answer can make a large dataset less useful than a smaller consistent one.

Deduplicate across training, development validation and final holdout boundaries. Keep all generated descendants of a grouped source on the intended side. If contamination is discovered, document the affected comparisons and rebuild an appropriate independent assessment; silently moving a single obvious duplicate does not establish that the related family is clean.

Do not claim a universal similarity cutoff or random split percentage prevents leakage. The useful test is whether cases share information that undermines the evaluation claim. Preserve the grouping and exclusion decisions with the frozen dataset version.

Step 6: Qualify the Dataset with an Original Case Review

Consider a proposed maintenance assistant that must produce complete handover summaries while preserving warnings. Its reference material contains approved, sanitized incident descriptions. The following examples illustrate review decisions, not generated output from a live run:

The desired training change is a reliable handover behavior, not memorization of invented incident facts. If source evidence is missing, do not ask the generator to fill gaps merely to produce a complete-looking record. Qualifying the example includes identifying when an unknown is the correct output.

Step 7: Freeze the Accepted Dataset and Its Limits

Create an immutable version of the accepted records, review evidence and provenance/grouping manifest. Recommended handoff information includes source permission checks, privacy processing, generation configuration, accepted/rejected counts with reasons, coverage, duplicate/leakage checks and the independent evaluation-set identity.

Have a dataset owner record unresolved gaps and the conditions under which the set may be reused. Permission changes, source-policy revisions or discovered systematic defects may require revisiting earlier decisions. A frozen file is a reproducibility aid, not proof that its contents remain suitable indefinitely.

The downstream training experiment must still compare the candidate against its baseline on independent tasks. No claim of improvement follows from accepted data alone. Artifact lineage, serving eligibility and post-training release operations belong in AI-300: Promoting and Operating Fine-Tuned Models.

Common Learning and Operational Mistakes

• Assuming readable source material is automatically permitted for generation and training.
• Treating synthetic output as anonymized or factually correct without inspection.
• Equating many paraphrases with diverse task coverage.
• Using the generator's own approval as the only quality check.
• Relying on a random row split while related scenarios cross partitions.
• Using the final holdout to repeatedly tune the generator or filtering rules.
• Accepting JSONL validity as proof of correct supervision or method compatibility.
• Turning a preview generation path into a claimed production guarantee.

Conclusion

Qualify synthetic training data through permission, privacy, constrained generation, substantive review and explicit leakage controls. Keep the final assessment independent and retain the evidence behind each accepted dataset version. The result is reviewed candidate supervision whose limits are visible, ready for a separate training and evaluation decision.

Authoritative Sources

• Fine-tuning in Microsoft Foundry
• Generate synthetic data for fine-tuning in Microsoft Foundry (preview)

Related Reading

• AI-300: Choosing Supervised, Preference, and Reinforcement Fine-Tuning
• AI-300: Promoting and Operating Fine-Tuned Models