Offline, real-time and voice simulation evals
In short
Evaluations score agent outputs against metrics: an LLM-as-a-judge rubric, a predefined RAG evaluator, or your own Python code. Datasets are versioned test data you build by hand, upload, or capture straight from production traces, and a run can either re-execute a workflow first or score what was already recorded. Beyond batch runs, a real-time eval attaches to a live deployment and samples its traffic continuously, and voice agents get their own simulation runs, scripted callers scored by an LLM judge before they take a real call.
What Evals does
- Three metric types
- Write an LLM-as-a-judge rubric, use a predefined RAG evaluator such as faithfulness, or score with your own Python function.
- Datasets from real traces
- Capture a production trace into a dataset with one click, so a real failure becomes a permanent regression test.
- Offline runs before you ship
- Re-run a candidate workflow version against a released dataset and compare its scores with what is in production today.
- Real-time evals on live traffic
- Attach a metric to a deployed App and score a sampled share of its traces automatically, with no re-run and no extra latency for callers.
- Voice simulations
- Run scripted callers against a deployed voice agent and have an LLM judge score pass rate, accuracy and experience.
- Every score traced back
- A run's results table links each score to the trace it came from, so a low score is one click from the evidence.
Datasets built from real traffic
A dataset is a versioned table of test items you build by typing rows in, uploading a JSON file, or capturing production traces directly from an App's Traces tab. Draft versions are editable; releasing one freezes it so evaluation runs stay reproducible against exactly the data they scored. Turning a bad trace into a dataset item is the highest-value path: when a real call goes wrong, capture it, and it becomes a regression test that catches the same failure again.
- Released versions are immutable; fork one when you need to change frozen data
- A trace-captured item carries input, output and status; With workflow mode scores it as recorded, not re-run
- Reshape captured items into flat rows to replay real inputs through a new workflow version instead
Real-time evals on a live deployment
An online evaluation attaches a saved metric to a deployed App from its Evaluations tab: pick a metric and version, a sample rate for what share of traffic to score, and an input path that pulls the metric's input out of each trace. Every finished trace becomes eligible as it arrives, sampled deterministically rather than by a random draw, so two evaluators at different rates never just score the same half of your traffic. Nothing is re-executed and callers see no difference; the score lands in the evaluator's own run list, one click from the trace that produced it.
- Start with a high sample rate while validating a new evaluator, then lower it on high-traffic Apps
- Failed and canceled runs are eligible too, not only successful ones
- An LLM-as-a-judge evaluator's reasoning, not just its score, is there for every run
Voice simulations
A voice agent gets a form of evaluation the rest of the platform does not: scenario sets, scripted callers defined by a persona, a goal and an expected outcome, run against the deployed agent and scored by an LLM judge. Run a set as text for a fast, cheap check of reasoning and tool use while iterating, or as voice to also exercise speech recognition and synthesis and get real latency percentiles before you ship a change.
- Generate starter scenarios from the agent's own configuration, then tighten the ones that matter
- Turn a real call that went wrong into a scenario directly from the Calls tab
- A run scores pass rate, accuracy and experience, with the judge's rationale for every scenario
Caller persona and goal
A patient who can only come after 4 pm and wants the earliest slot this week.
Expected outcome
The agent offers a slot after 4 pm and confirms it without transferring the call.
Predefined evaluators
- AnswerCorrectness
- ContextPrecision
- ContextRecall
- FactualCorrectness
- Faithfulness
Where teams use it
- AI agent governanceScreen what reaches the model, approve what the agent does, and keep a replayable record of every run, with the same controls on every agent.
- Contact center voice agentsAnswer your phone lines with voice agents that look up the caller's account, resolve routine calls and transfer to a person when a call needs one.
- Customer serviceAnswer inbound calls and chats, look up and act on the account, and hand complex cases to a person, with every conversation traced.
Works with
- ObservabilityEvery run is traced and replayable, node by node, with the cost, latency and tokens each step used.
- Voice AgentsVoice Agents answer calls on the phone, the web or your own backend, reason with an LLM, and speak back, with every turn measured.
- Apps and deploymentsDeploy a saved workflow version as an App with its own hostname. Call it over HTTP, embed a chat widget, or trigger it on a schedule or an event.
Questions and answers
What is the difference between an evaluation run and a real-time eval?
An evaluation run scores a fixed, released dataset on demand, useful for comparing a candidate workflow version against production before you ship. A real-time eval instead attaches to a live deployment and scores a sample of its actual traffic continuously, after you ship.
Can I turn a real production failure into a test case?
Yes. Open the trace from an App's Traces tab and add it to a dataset in one click; the input, output and status become a dataset item you can score or use as a regression test.
What is an LLM-as-a-judge metric?
A metric where an LLM follows a rubric you write and returns a score, suited to subjective qualities such as hallucination, relevance or tone that a fixed rule cannot capture.
How are voice agents evaluated differently?
Through simulations: scripted callers with a persona and an expected outcome run against the deployed agent, in text for fast iteration or in voice to also exercise speech recognition and synthesis, and an LLM judge scores the result.
Does a real-time eval slow down my App?
No. Traces are scored after the fact from what was already recorded; nothing re-executes and callers see no added latency.
See an agent on your own workflow.
Bring a process and its documents. Our engineers will show you how Dynamiq runs it, in your environment or ours.