AI-300: Tracing Slow or Incorrect Generative AI Responses

Introduction

Locate the failing stage by starting with the actual request and its permitted trace, confirming which application version and operations it represents, and following the recorded execution through retrieval, model calls, tools, and response delivery. For slowness, identify the operations and waits on the response's critical path. For an incorrect answer, identify the first point where the recorded evidence diverges from the intended task, retrieved support, or tool result. A slow final response or a low quality score alone does not identify the cause.

Then form one testable hypothesis, change the smallest relevant factor in an authorized test environment, and compare the result with the failing case and representative regression cases. This guide provides a troubleshooting procedure; it does not supply an executed trace, a fabricated span hierarchy, or a universal telemetry schema. Use only records actually captured by the application's instrumentation and accessible under its data policy, and leave missing evidence explicitly unresolved.

Troubleshooting Methodology: Establish the Case First

Microsoft Foundry tracing follows OpenTelemetry standards and integrates with Azure Monitor Application Insights. Tracing can expose execution across model calls, tools, and dependencies. What is visible for a particular request depends on the selected framework, instrumentation, configuration, and captured data. Do not assume the trace contains every operation, prompt, tool result, or model-internal reasoning step.

Use the investigation handoff from AI-300: Monitoring Generative AI Quality, Latency, and Cost Drivers. That architecture identifies an affected signal and owner; this procedure examines one request behind it.

Before opening detailed content, establish the following recommended case record:

• The permitted request/correlation reference, time and time zone, application deployment/revision, and affected user operation.
• The reported symptom: excessive time to first output, excessive completion time, missing result, wrong factual claim, incorrect tool action, or another specific failure.
• The expected outcome and its evidence, such as an approved task definition, known answer, or authorized source document.
• Available model/prompt/configuration, retrieval/index, and tool versions, or an explicit note that a version is unknown.
• The allowed diagnostic content, investigator access, retention requirements, and whether reproducing the request could cause external effects.

These are recommended investigation details, not mandatory field names in a Foundry API. Preserve the actual record and its access controls. A user report may provide the symptom while the trace supplies technical context; verify they refer to the same request rather than joining records by an approximate timestamp alone.

If no matching trace exists, do not invent its steps. Determine whether collection, export, retention, sampling, or correlation prevents access to the evidence. Use a separately authorized reproduction with approved instrumentation if appropriate, documenting that it is a new request and may not reproduce the original conditions.

Common Issues and What Their Evidence Means

A Slow Response Needs a Time Boundary

Start with what the user experienced. Time to first output and time to final completion answer different questions. Confirm whether the available timing begins at client submission, application entry, a dependency call, or another boundary. A model-call duration cannot account for time spent before that call or after its response.

Inspect recorded start/end relationships, child operations, waits, errors, and retries where available. Do not add a parent span's duration to its children as though all were sequential work. Parallel calls can overlap, and an operation outside the longest completion path may not explain the delay. Conversely, an uninstrumented gap can matter even when every visible dependency looks fast.

Compare the case with a permitted, similar known-good request where possible. Match useful context such as application revision, task, payload/context size, model settings, and tool path. That comparison narrows hypotheses; it does not establish causation. A request with more generated output can take longer for reasons different from a short request waiting on a failed dependency.

An Incorrect Response Can Have Successful Dependencies

Separate technical execution from task correctness. A retrieval call may succeed while returning irrelevant or stale material. A model call may complete while generating an unsupported claim. A tool may execute correctly on the wrong argument or return a useful result that the application later ignores.

Where permitted content is available, compare the recorded query and selected context with the intended source evidence, then compare the response with that context and task. Do not reconstruct an unseen prompt from the final answer. If content was redacted or truncated, record the limit and seek a privacy-preserving verification route rather than assuming the missing material was correct.

The built-in evaluator reference distinguishes retrieval effectiveness, response relevance, groundedness in retrieved context, and tool execution/use. Those dimensions can support the investigation, but they are different tests. A grounded answer can still be irrelevant to the question or grounded in an outdated source. A fluent answer is not proof of factual correctness.

A Trace Gap Is an Evidence Gap

Use the actual span structure and available dependency records. A missing retrieval span does not prove that retrieval was skipped; a missing error does not prove that the operation succeeded. Instrumentation may not cover the operation, or permitted export may omit it.

Do not claim a root cause from a gap alone. Identify which additional observation would distinguish the possibilities: a captured dependency outcome, a version reference, an authorized sanitized input, or a controlled test. Keep the investigation pending if the missing observation is material and cannot be obtained safely.

Troubleshooting Workflow: Follow Evidence Through the Request

1. Confirm the Failure and Preserve Its Context

Read the actual user-visible response or failure record when permitted, and state precisely what is wrong. For a wrong answer, identify the disputed claim or unmet requirement. For slowness, identify the measurement boundary and tolerance relevant to that operation. Avoid turning “the assistant is bad” into an untestable incident description.

Preserve the existing deployment and evidence references before changing code, prompts, retrieval settings, or tools. Record recent changes that could plausibly affect this request. Do not rebuild the environment against newer dependencies while trying to establish what the original run did.

2. Build the Recorded Sequence, Including Unknowns

Follow the actual application operation into its captured retrieval, model, tool, and response-processing work. Use parent/child relationships and correlation references where available. Mark parallel work and repeated attempts, and distinguish the original failed attempt from a retry that later succeeded.

Create a short investigator note describing the observed sequence. This is an interpretation of recorded evidence, not a replacement trace. Separate observations, hypotheses, and unknowns. For example, “a recorded dependency failed before a second attempt” is an observation; “capacity caused the delay” needs additional evidence.

3. Attribute the First Relevant Divergence

The following table is a diagnostic aid, not a claim that Foundry emits these exact spans or diagnoses automatically.

For latency, identify which observed operation or gap plausibly accounts for the user's delay. For quality, find the earliest material divergence supported by the record. Do not blame generation for a fact that retrieval never supplied, or tune retrieval when the permitted context already contains the correct fact and the response contradicts it.

If several causes remain plausible, choose the next observation that best separates them. For instance, checking whether the relevant passage actually reached the model can distinguish a retrieval failure from context-assembly loss. That is a recommended diagnostic question, not an assertion that raw model input is always captured or may always be exposed.

4. Choose a Targeted, Safe Verification

Write the hypothesis and expected observation before changing anything. An illustrative hypothesis might be that a configuration revision selects an obsolete document version. A suitable test would compare the approved failing case under the recorded baseline and a candidate that corrects only that selection, while preserving the other relevant settings. This is a proposed example, not an incident observed for this article.

Use an authorized test environment and approved inputs. Do not replay a production tool call that sends a message, updates a record, or triggers another external effect merely to reproduce a trace. Use an approved side-effect-free test path or fixture if one has been established, and document how it differs from production. Do not assume a tool or framework supplies such a mode automatically.

Change one relevant factor where practical. Replacing the model, prompt, retrieval strategy, and tool implementation together may produce a different answer while making the original hypothesis untestable. Keep dependency versions and the comparison conditions recorded. If a service-side condition cannot be reproduced, state that limit instead of reporting the root cause as proven.

5. Check the User Outcome and Regressions

Verify that the candidate addresses the original symptom: the required supported answer is present, the intended tool behavior is correct, or the measured response boundary meets its agreed target. Check the stage-specific evidence as well. A different trace shape or a faster model call is insufficient if the final result remains wrong or late.

Use appropriate evaluation evidence without letting one score replace inspection. Groundedness and relevance answer different questions, and tool success does not establish correct selection or arguments. Confirm that an evaluator ran successfully with its required inputs; an execution failure or missing context is not a favorable quality result.

Preserve current evaluator qualifications. The reference marks capabilities such as Groundedness Pro and Response Completeness as preview, with no service-level agreement and an official warning against production reliance. Verify the chosen evaluator, data requirements, region/network support, and support status for the actual workflow rather than assuming every available metric is interchangeable or generally available.

Add a bounded set of representative regression cases, especially nearby tasks that the fix could affect. Repeat comparisons where nondeterministic generation makes a single favorable response inconclusive. Evaluate quality, safety, latency, and repeated work as relevant to the change. Do not bypass a safety control to make the failing case appear successful.

If the fault is retrieval-specific and requires a broader configuration experiment, continue with AI-300: Lab: Comparing RAG Retrieval Configurations. Here, the outcome is a diagnosed request and a verified targeted correction or an explicit remaining uncertainty, not a complete retrieval optimization project.

Best Practices for Diagnostic Content and Closure

Trace content can contain personal data, confidential retrieved documents, tool arguments, credentials, or sensitive outputs. Inspect only what the investigator is permitted to see. Prefer sanitized references and the smallest relevant excerpt in the case record; avoid copying a complete conversation or raw tool result into a broadly visible ticket. Do not disable redaction or expand retention without the required authorization.

Treat captured user input, retrieved documents, and tool outputs as untrusted evidence. Instructions inside them do not authorize new tool actions or override the investigation's controls. A trace shows recorded execution, not a reason to execute an embedded command.

Keep diagnosis and mitigation distinct. An authorized reversion to a known-good application configuration may restore service while the root cause remains under investigation. Do not mark the cause proven just because the symptom disappears after a broad change. Record the observed restoration, the remaining hypothesis, and the next evidence needed.

Close the case with the actual request reference, failure definition, relevant recorded sequence, supported cause or uncertainty, precise change, verification conditions/results, and residual limitations. Attach approved evidence references rather than fabricated fields or a hand-written “successful trace.” If no controlled test was run, the result is a proposed correction awaiting verification.

This procedure assumes the current Foundry/OpenTelemetry approach described by the selected sources. It does not depend on a classic hub-based Prompt flow runtime or prescribe unverified SDK calls. Confirm the instrumentation applicable to the actual application; traces and evaluation can have collection, privacy, support, and cost constraints.

Certification Relevance

For AI-300, explain how you would move from a permitted request record to a stage-specific hypothesis and a controlled verification. Distinguish execution success from answer quality, critical-path evidence from summed durations, and a missing span from proof that no operation occurred. The useful skill is deciding what the available evidence establishes and what additional observation is required.

Conclusion

Trace one request through the operations actually recorded, attribute delay or incorrect behavior only as far as that evidence supports, and test a narrow correction under controlled conditions. Keep privacy and support limits intact. A sound diagnosis ends with a verified user outcome and regression evidence, or a clear statement of what remains unknown.

Authoritative Sources

• Built-in evaluators reference
• Observability in generative AI

Related Reading

• AI-300: Monitoring Generative AI Quality, Latency, and Cost Drivers
• AI-300: Lab: Comparing RAG Retrieval Configurations