Metrics & Tracing
Understand dashboard metrics and inspect per-message traces for reliability, latency, feedback, and credit usage.
The assistant dashboard shows how fast, reliable, and cost-efficient your assistant is. Tracing lets you inspect an individual message to see its LLM calls, tool calls, token usage, latency, and credits.
Use metrics to spot trends. Use a trace to find the cause of a slow, failed, or expensive turn.
Dashboard metrics
Go to Admin Panel → Assistants → Dashboard. The date range at the top applies to period metrics and charts. For operational metrics, select Chat, Standard Voice, or Realtime Voice. The analytics requests use the same date range, browser timezone, and selected mode.
The range includes both its start and end dates in the browser timezone. Two summary cards are exceptions: Active Today always uses the current local day, while Memories is the assistant's current total.
Mode-scoped metrics should only be compared within the same mode because Chat and Voice use different latency milestones. The selected mode and date range are also stored in the URL, so the same dashboard view can be reloaded or shared.
Reading percentiles
| Term | Meaning |
|---|---|
| P95 | 95% of samples are at or below this value. Use it to identify slow tail experiences. |
— means there is not enough data or the metric does not apply. It does not mean zero latency.
Operational metrics by mode
| Metric | Definition | How to use it |
|---|---|---|
| P95 Response | P95 response-start latency over completed turns with valid timestamps | Identify slow tail experiences. The start milestone depends on Chat or Voice. |
| Technical Failure Rate | failed / (completed + failed) | Tracks technical pipeline failures. User-interrupted turns are not counted as technical failures. |
| Cost per Completed Turn | Attributed credits divided by completed turns | Compare cost across models, date ranges, and interaction modes. |
| Tool Error Rate | Tool executions with action ERROR divided by all tool executions | Track tool-loop reliability for Chat and Standard Voice. |
Cost per Completed Turn can be — for Realtime Voice because that mode is session-priced. Tool Error Rate is also — for Realtime Voice until provider-owned tool loops emit canonical tool executions.
The summary cards also display Positive Feedback Rate and LLM Trace Issue Rate for the selected date range. Positive Feedback Rate is — when the period has no ratings. LLM Trace Issue Rate is the percentage of LLM calls with an error or aborted status. Do not use it as a substitute for Technical Failure Rate: a user interruption can produce an aborted trace even when the pipeline did not fail technically.
Chat latency
| Metric | Starts | Ends |
|---|---|---|
| First Response Latency | Server receives the chat request | First response text is sent to the user |
First Response Latency is the end-to-end latency experienced by the user. It can include context loading, memory retrieval, and tool loops before the first response appears. It is different from LLM TTFT, which covers only one LLM call.
Voice latency
| Metric | Starts | Ends |
|---|---|---|
| End-of-Speech → First Audio Sent | The system determines that the user has finished speaking | Server sends the first audio frame to the client/device |
| Endpointing Latency | User finishes speaking | Audio input is committed for processing |
| ASR Latency | Audio input is committed | Final transcript is available |
| Agent Latency | Final transcript is available | First response text is available |
| TTS Text Buffering | First response text is available | First TTS request is sent |
| TTS First Audio Latency | First TTS request is sent | Provider returns the first audio |
| Audio Send Latency | Provider returns the first audio | Server sends the first audio frame |
For Standard Voice, End-of-Speech → First Audio Sent covers the complete endpointing → ASR → Agent → TTS → audio-send pipeline. The stage metrics identify the specific bottleneck.
For Realtime Voice, the provider handles audio in an integrated pipeline. The dashboard reports only the P95 End-of-Speech → First Audio Sent metric; ASR/Agent/TTS stages are — instead of synthetic estimates. Status and duration remain available in individual turn details.
End-of-Speech → First Audio Sent and Audio Send Latency end when the server sends the first frame. They do not measure when the user actually hears audio; network transit, client buffering, and speaker playback can add latency.
What Is Tracing?
A single assistant reply is not always one LLM call. When the assistant uses tools (knowledge base, MCP, custom APIs, etc.), it runs a tool loop: call the model → run the requested tools → feed results back → call the model again, until it has a final answer.
Tracing records each step of that loop:
- One LLM call entry per loop iteration: model, provider, tokens, duration, and status.
- One tool execution entry per invoked tool: arguments, result, duration, and outcome.
This is built-in observability — no external setup is required.
How to Open Traces
- Go to the Admin Panel and open Assistants.
- Open an assistant, then go to Conversations.
- Select a conversation to view its messages.
- On any assistant message, click the Trace pill to open Agent Trace.
Only assistant messages have traces. User messages do not trigger LLM or tool calls, so there is nothing to trace.
Turn Summary
The top of the dialog shows the turn's interaction mode, channel, Voice mode when applicable, and status. Technical badges use labels such as Completed, Web app, Manual, Standard, and Realtime. The summary shows First Response Latency for Chat or End-of-Speech → First Audio Sent for Voice, plus total attributed credits. Voice turns show Voice Latency Breakdown when stage timings are available.
The timeline then combines LLM calls and charged usage steps such as ASR, TTS, memory, or Realtime Voice in execution order. Credits and their attribution type appear on the relevant step.
Reading an LLM Call
Each LLM call appears as a timeline card. The card identifies its phase, such as a direct response, tool call, or result synthesis, and shows the model, status, input/output tokens, duration, and tool count. Expand it for provider, total/cached/reasoning tokens, TTFT, cost, and timestamp.
| Field | Meaning |
|---|---|
| Model | Chat model that handled the call, for example gpt-5.6-luna |
| Provider | Provider behind the model, for example openai or anthropic |
| Status | completed, aborted, or error |
| Time | When the call started |
| Duration | How long the call ran |
Token usage
| Metric | Meaning |
|---|---|
| Input tokens | Tokens sent to the model: prompt, history, context, and tool results |
| Output tokens | Tokens generated and reported by the provider |
| Total tokens | Total tokens reported by the provider for the LLM call |
| Cached tokens | Input tokens read from the provider's prompt cache |
| Reasoning tokens | Tokens used for internal reasoning when reported by the provider/model |
Cached tokens and Reasoning tokens appear only when greater than zero. Token accounting can differ by provider, so compare usage within the same model and provider.
LLM latency
| Metric | Meaning |
|---|---|
| Duration | Total time from sending an LLM request until its stream completes or stops |
| TTFT (Time to First Token) | Time from the start of one LLM call to its first content token; it excludes stages before that call |
An answer with multiple tool loops has multiple LLM calls and multiple TTFT/Duration values. Do not add TTFT values together as a replacement for First Response Latency.
Tool Executions
When the assistant uses tools, the call card lists every execution. Each row shows the tool name, a type badge, and its duration. Expand a row for details.
| Detail | Meaning |
|---|---|
| Action | Outcome of the tool call |
| Time | When the tool ran |
| Args | Arguments passed by the model, in JSON |
| Result | Summary of the tool result |
| Error | Error message when the tool failed |
Tool types
| Badge | Meaning |
|---|---|
| builtin | Server-side built-in tools such as knowledge base, memory, and web |
| mcp | Tools provided through MCP endpoints and servers |
| custom | Your custom HTTP API tools |
Tool actions
| Action | Meaning |
|---|---|
| REQLLM | Result was sent back to the model to continue the loop |
| RESPONSE | Tool produced the final response directly |
| NONE | Tool ran but returned nothing requiring further action |
| ERROR | Tool failed; inspect the error message |
Args and Result are sanitized and truncated for safety and readability. They are summaries, not complete raw payloads.
Trace Status
| Status | Meaning |
|---|---|
| completed | Call completed normally |
| aborted | Call stopped early, for example when the user interrupted or barged in |
| error | Call failed; inspect the trace and tool errors for the cause |
Common Use Cases
Debug a wrong answer — inspect which tools were called, their arguments, and their results. A bad result or incorrect arguments often explain an off-topic reply.
Analyze cost — token usage maps to credit consumption. Use traces to find expensive messages and understand why. See Credits.
Diagnose a slow reply — compare each loop and tool duration. A slow external tool or too many tool loops is often the bottleneck.
Related
- Tools & MCP — tools that appear in traces
- AI Models — models shown in each LLM call
- Credits — how token usage maps to cost
- Standard vs Realtime — why voice modes expose different metrics