Skip to content

Commit 82c1305

Browse files
scott-lowe-vapiclaude
scott-lowe-vapi
and
claude
committed
TEST-202 docs: add the assistant Test tab page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 8c6bd27 commit 82c1305

3 files changed

Lines changed: 94 additions & 0 deletions

File tree

‎fern/docs.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -689,6 +689,9 @@ navigation:
689689
- page: Quickstart
690690
path: observability/simulations-quickstart.mdx
691691
icon: fa-light fa-rocket
692+
- page: Test an assistant
693+
path: observability/simulations-test-tab.mdx
694+
icon: fa-light fa-vial
692695
- page: Advanced
693696
path: observability/simulations-advanced.mdx
694697
icon: fa-light fa-flask-vial

‎fern/observability/simulations-quickstart.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ slug: observability/simulations-quickstart
77

88
In about 10 minutes, you'll run an AI tester through an appointment-booking conversation with Riley and get a pass or fail result. You'll also have a reusable simulation suite that you can run again after any change.
99

10+
Testing a single assistant? Start from its [**Test tab**](/observability/simulations-test-tab), where Composer can generate a suite for you.
11+
1012
## What you'll build
1113

1214
A reusable **simulation suite** that asks an AI tester to book an appointment with Riley, Vapi's appointment-scheduler template assistant. The first criterion checks whether Riley reports the requested date and time with a confirmation number. This is a conversation-only demonstration, not proof that an appointment exists in a calendar.

‎fern/observability/simulations-test-tab.mdx‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
title: Test an assistant
3+
subtitle: Generate, run, and track simulation suites from an assistant's Test tab
4+
description: "Use an assistant's Test tab to generate a simulation suite with Composer, run it, and follow its latest results and run history without leaving the assistant."
5+
slug: observability/simulations-test-tab
6+
---
7+
8+
The **Test** tab on an assistant lists the [**simulation suites**](/observability/simulations-overview) assigned to it. From the tab, you can generate a suite with Composer, run or rerun a suite, and open its results and run history without leaving the assistant.
9+
10+
## Prerequisites
11+
12+
- An [**assistant**](/assistants) in your Vapi account.
13+
- Permission to read test suites and test runs. Without both, the **Test** tab does not appear.
14+
- Permission to create test suites to generate a suite, and permission to create test runs to run one.
15+
16+
<Note>
17+
Runs use the published version of the assistant. If you're editing a draft, the tab shows **Runs use the published version**. Publish your changes to test them. If [**traffic splitting**](/assistants/versioning/traffic-splitting) is enabled, publish the new version with 0% traffic so it handles no live calls.
18+
</Note>
19+
20+
## Generate a suite with Composer
21+
22+
Open the assistant and select the **Test** tab. If no suites are assigned to the assistant, the tab shows **No test suites yet** with **Generate with Composer** and an **About simulations** link. Once the assistant has suites, select **Generate suite** above the table to create another.
23+
24+
{/* Screenshot: Test tab empty state with "No test suites yet" and the Generate with Composer button */}
25+
26+
Both buttons open [**Composer**](/composer#generate-tests-for-an-assistant) in a new thread for this assistant:
27+
28+
<Steps>
29+
<Step title="Review the test requirements">
30+
Composer reads the assistant's prompt and tools and drafts **Test requirements**. Select **Generate coverage** to accept them, or type the changes you want in the chat.
31+
</Step>
32+
33+
<Step title="Review the generated suite">
34+
Composer generates simulations from the requirements and saves them as a suite assigned to the assistant. The suite card shows **Saved to the Test tab**. Tools that reach your systems are mocked by default, and each simulation lists any tools it mocks.
35+
</Step>
36+
37+
<Step title="Run the suite">
38+
Select **Run suite** on the card to start a chat run with one iteration. The suite also appears in the **Test** tab.
39+
</Step>
40+
</Steps>
41+
42+
## Read the suite table
43+
44+
The table shows up to 100 suites assigned to the assistant, one row per suite, with the latest run of that suite against this assistant.
45+
46+
{/* Screenshot: Test tab suite table with Status, Progress, and Last run columns */}
47+
48+
| Column | What it shows |
49+
| -- | -- |
50+
| **Test suites assigned** | The suite name. Once the suite has run, select the row to open its latest run. |
51+
| **Simulations** | How many simulations the latest run executed, with its chat or voice mode. Before the first run, the number of simulations in the suite. |
52+
| **Status** | The latest run's status: **Queued**, **Running**, **Passed**, **Failed**, **Errored**, or **Canceled**. A suite that has never run for this assistant shows **Not run yet**. |
53+
| **Progress** | A bar of the latest run's outcomes. During a run, it shows how many simulations are complete, such as `2 of 5 complete`. After the run, it summarizes the outcomes, such as `4 passed · 1 failed or errored`. |
54+
| **Last run** | When the latest run ended, such as `5 minutes ago`. It updates every minute while the tab is open. |
55+
56+
Each row ends with a **Run** or **Rerun** button. Once a suite has run, the row also shows a run history button that opens **Simulations** > **Runs** filtered to that suite and this assistant. To see every run for the assistant, select **View run history** above the table.
57+
58+
## Run a suite
59+
60+
Select **Run** on a suite that has never run for this assistant. If the suite can run in chat, it starts immediately as a chat run with one iteration, and the row updates as it progresses.
61+
62+
The **Run suite** dialog opens instead, so you can choose the mode and iterations, when:
63+
64+
- The assistant needs voice, such as one that uses a GPT Live model. **Voice** is the only mode offered.
65+
- One of the suite's simulations is missing.
66+
- A scenario has hooks configured.
67+
68+
To run a suite again, select **Rerun**. The dialog always opens so you can choose **Chat** or **Voice** and the number of **Iterations**. It starts with the mode of the previous run. Each rerun creates a new run and picks up any changes made to the suite.
69+
70+
<Note>
71+
A suite with no simulations can't run. Add at least one simulation before running it. Run and Rerun stay disabled while a run of the suite is queued or running.
72+
</Note>
73+
74+
## Next steps
75+
76+
<CardGroup cols={2}>
77+
<Card title="Simulations quickstart" icon="rocket" href="/observability/simulations-quickstart">
78+
Build and run a simulation suite step by step.
79+
</Card>
80+
<Card title="Manage simulations" icon="sliders" href="/observability/simulations-manage">
81+
Edit suites, review results, and maintain coverage.
82+
</Card>
83+
<Card title="Mock tool responses" icon="flask" href="/observability/simulations-advanced#mock-tool-responses">
84+
Control what tools return during a simulation.
85+
</Card>
86+
<Card title="Composer" icon="wand-magic-sparkles" href="/composer#generate-tests-for-an-assistant">
87+
Generate tests for an assistant in conversation.
88+
</Card>
89+
</CardGroup>

0 commit comments

Comments
 (0)