Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions api-reference/asyncapi-asr-realtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -93,15 +93,15 @@ channels:
enum:
- transcribe-stream-1
default: transcribe-stream-1
description: Realtime speech-to-text model. Optional; `transcribe-stream-1` is the default and the only realtime model. The legacy name `transcribe-1-pro-realtime` reaches the same model.
description: Realtime speech-to-text model. Optional; `transcribe-stream-1` (Preview) is the default and the only realtime model. The legacy name `transcribe-1-pro-realtime` reaches the same model.
description: |
Realtime speech-to-text channel. Clients stream audio and receive
transcripts turn by turn, with turn detection, optional live drafts and
conversation context.

## Connection Headers
- `Authorization: Bearer <api_key>` - Required for authentication (see security section)
- `model: transcribe-stream-1` - Optional; the default
- `model: transcribe-stream-1` (Preview) - Optional; the default

Every message in both directions is one MessagePack map in a binary
frame, with a `type` field. Send `config` first, wait for
Expand Down
5 changes: 4 additions & 1 deletion api-reference/endpoint/websocket/asr-realtime.mdx
Original file line number Diff line number Diff line change
@@ -1,23 +1,26 @@
---
asyncapi: "/api-reference/asyncapi-asr-realtime.yml asrRealtime"

Check warning on line 2 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L2

Did you really mean 'asrRealtime'?
title: "WebSocket Realtime Speech to Text"

Check warning on line 3 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L3

Did you really mean 'Realtime'?
description: "Stream audio over a WebSocket and receive transcripts turn by turn, with turn detection, live drafts and conversation context"
icon: "webhook"
iconType: "solid"
tag: "Preview"
---

<Note>
Connect to `wss://api.fish.audio/v1/asr/realtime` with a WebSocket client.
Every message in both directions is one MessagePack map in a binary frame. A
normal HTTP GET request does not start a session. For an introduction, see
[Realtime Speech to Text](/features/speech-to-text/realtime).

Check warning on line 14 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L14

Did you really mean 'Realtime'?
</Note>

<Note>`transcribe-stream-1` is in preview.</Note>

## Before you start

You need a [Fish Audio API key](/developer-guide/getting-started/api-key) and a WebSocket client that can set request headers. The API key travels in a request header, so connect from your server, not from a browser; to capture audio in a browser, relay it through your server as in [Transcribe a live audio stream](/developer-guide/sdk-guide/cookbook/realtime-transcription#capture-from-a-browser-microphone). The examples on these pages use Python with `websockets` and `msgpack`, and Node.js with `ws` and `@msgpack/msgpack`.

This endpoint is separate from the realtime endpoints of the [compatible APIs](/developer-guide/compat/capabilities), which use manual commit and have no server-side turn detection.

Check warning on line 23 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L23

Did you really mean 'realtime'?

## Connect

Expand All @@ -30,11 +33,11 @@
| Header | Required | Description |
| --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | Yes | `Bearer <API key>`. Write the scheme exactly as `Bearer`, followed by a space and the key. |
| `model` | No | `transcribe-stream-1`, the default and the only realtime model. The legacy name `transcribe-1-pro-realtime` reaches the same model. |
| `model` | No | `transcribe-stream-1` (Preview), the default and the only realtime model. The legacy name `transcribe-1-pro-realtime` reaches the same model. |

Check warning on line 36 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L36

Did you really mean 'realtime'?

Each open connection counts against the same concurrency limit as your other API requests, from the moment it opens until it closes, even while you send no audio. See [Concurrent request limits](/developer-guide/models-pricing/pricing-and-rate-limits#concurrent-request-limits). A session cannot be resumed: a new connection is a new session.

Sessions are billed per second of session duration, as Realtime ASR. See [ASR pricing](/developer-guide/models-pricing/pricing-and-rate-limits#automatic-speech-recognition-asr-models).

Check warning on line 40 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L40

Did you really mean 'Realtime'?

If the server refuses the upgrade, it answers with a plain HTTP response whose JSON body is `{"status": <int>, "message": <str>}`:

Expand All @@ -42,7 +45,7 @@
| ------ | ----------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| 401 | The API key is missing or invalid, or the `Authorization` header does not start with `Bearer `. | Check the key and the header. |
| 402 | Insufficient API credit. | Top up your balance. |
| 429 | Your team is at its concurrency limit, or the service is at capacity. | Retry with backoff. Honor `Retry-After` when present. |

Check warning on line 48 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L48

Did you really mean 'backoff'?
| 503 | The service is restarting. | Retry. |

The `101 Switching Protocols` response carries an `x-request-id` header, and so do the capacity 429 and 503 refusals. It is the same value as `request_id` in `session_started`, `error` and `session_done`. Log it, and include it when you contact support. Refusals for the key, balance and your concurrency limit (401, 402 and concurrency 429) happen before a session is assigned and carry no `x-request-id`.
Expand Down Expand Up @@ -89,13 +92,13 @@
{
"type": "config",
"audio": { "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 },
"turn_detection": { "mode": "auto", "eager": true },

Check warning on line 95 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L95

Did you really mean 'turn_detection'?
"results": { "drafts": true, "ignore_timestamps": false },
"context": {
"history": [{ "role": "assistant", "text": "Hi, how can I help?" }],
"terms": ["Fish Audio", "transcribe-stream-1"]
},
"client_reference_id": "call-8f2c"

Check warning on line 101 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L101

Did you really mean 'client_reference_id'?
}
```

Expand Down Expand Up @@ -321,7 +324,7 @@
| `turn_index` | integer | The open turn, or, if no turn is open, the index the next turn will get. |
| `audio_ms` | integer | Where the acknowledgement is in the audio. |

`auto` mode only, while your agent is speaking (see [`agent_speaking`](#agent_speaking)): the user acknowledged ("mm-hm", "yeah", "right") without taking the turn. Keep talking. One message per acknowledgement. It always comes before the named turn's `committed`, and it can come before that turn's `speech_start`. A backchannel counts for the turn that contains it; that turn commits with `user_turn: false` and `turn_kind: "backchannel"` unless the user goes on to take the turn. A turn that opened in `vad` or `manual` mode never counts an acknowledgement. When your agent is not speaking, an acknowledgement takes the turn (`user_turn_taken`).

Check warning on line 327 in api-reference/endpoint/websocket/asr-realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

api-reference/endpoint/websocket/asr-realtime.mdx#L327

Did you really mean 'backchannel'?

### `committed`

Expand Down
2 changes: 1 addition & 1 deletion developer-guide/models-pricing/models-overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,13 +71,13 @@

## Speech-to-Text Models

We recommend `transcribe-1-pro` for transcribing recordings. Both file models use `POST /v1/asr`, and you select one with the `model` HTTP header, written exactly as shown. Send `model: transcribe-1-pro` with every request: if the header is missing or its value is not an exact match, the request is served and billed as `transcribe-1`, and no error is returned. To transcribe live audio, use `transcribe-stream-1` on the realtime WebSocket, `wss://api.fish.audio/v1/asr/realtime`, where it is the default model.

Check warning on line 74 in developer-guide/models-pricing/models-overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/models-overview.mdx#L74

Did you really mean 'realtime'?

| Model | Capabilities |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `transcribe-1-pro` (recommended) | Anything from short clips to long recordings (up to 60 minutes), including multi-speaker conversations. Speaker markers in `text`, structured `speaker_turns` with start and end times, speaker-count hints, and inline emotion and vocal-event cues such as `[高兴]` (happy) and `[laughter]` (optional via `tag_audio_events`). Supports optional timestamps. |
| `transcribe-1` | General transcription of short recordings, with optional timestamps. Serves requests whose `model` header is missing or not an exact match. |
| `transcribe-stream-1` | Live audio streamed over the [realtime WebSocket](/features/speech-to-text/realtime), transcribed turn by turn while people speak. Turn detection for voice agents (eager end of turn, backchannel detection and background awareness), live drafts, conversation context, speaker tags and sound tags. Sessions up to 60 minutes. |
| `transcribe-stream-1` (Preview) | Live audio streamed over the [realtime WebSocket](/features/speech-to-text/realtime), transcribed turn by turn while people speak. Turn detection for voice agents (eager end of turn, backchannel detection and background awareness), live drafts, conversation context, speaker tags and sound tags. Sessions up to 60 minutes. |

Check warning on line 80 in developer-guide/models-pricing/models-overview.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/models-overview.mdx#L80

Did you really mean 'backchannel'?

Pro labels speakers with inline `<|speaker:N|>` markers in `text`, alongside emotion cues. A label identifies the same speaker within one response only. When you request timestamps (`ignore_timestamps=false`), Pro also returns `speaker_turns`, a structured list of turns with each turn's speaker, text, and start and end times, unless you send `diarize=false`. Timestamped `segments` are word-level, contain spoken words without these markers, and do not expose per-segment speaker IDs. See the [multi-speaker response example](/features/speech-to-text#multi-speaker-conversations) and [Speaker turns](/features/speech-to-text#speaker-turns).

Expand Down
10 changes: 5 additions & 5 deletions developer-guide/models-pricing/pricing-and-rate-limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,15 @@

### Automatic Speech Recognition (ASR) Models

| Model Name | Price (USD) |
| ------------------------------------ | -------------------- |
| `transcribe-1-pro` (recommended) | $0.36 / audio hour |
| `transcribe-1` | $0.36 / audio hour |
| `transcribe-stream-1` (Realtime ASR) | $0.36 / session hour |
| Model Name | Price (USD) |
| --------------------------------------------- | -------------------- |
| `transcribe-1-pro` (recommended) | $0.36 / audio hour |
| `transcribe-1` | $0.36 / audio hour |
| `transcribe-stream-1` (Realtime ASR, Preview) | $0.36 / session hour |

Check warning on line 46 in developer-guide/models-pricing/pricing-and-rate-limits.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/pricing-and-rate-limits.mdx#L46

Did you really mean 'Realtime'?

For recordings, we recommend `transcribe-1-pro`, which handles multi-speaker conversations (speaker markers and speaker turns), long recordings, and inline emotion cues. Both file models cost the same and use the [Speech to Text API](/api-reference/endpoint/openapi-v1/speech-to-text). Send the `model: transcribe-1-pro` HTTP header to select Pro; a request whose `model` header is missing or not an exact match is served and billed as `transcribe-1`. See the [model comparison](/features/speech-to-text#choose-an-asr-model).

`transcribe-stream-1` transcribes live audio over the [realtime WebSocket](/features/speech-to-text/realtime) and is billed as its own line item, Realtime ASR, separate from file transcription.

Check warning on line 50 in developer-guide/models-pricing/pricing-and-rate-limits.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/pricing-and-rate-limits.mdx#L50

Did you really mean 'Realtime'?

**How ASR billing works for files (`transcribe-1-pro`, `transcribe-1`):**

Expand All @@ -56,7 +56,7 @@
- The full duration of the recording is billed, including silence, once per successful request
- Requests that return an error response are not billed

**How Realtime ASR billing works (`transcribe-stream-1`):**

Check warning on line 59 in developer-guide/models-pricing/pricing-and-rate-limits.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/pricing-and-rate-limits.mdx#L59

Did you really mean 'Realtime'?

- Charges are based on session duration, billed per second, once per session
- Session duration is the time the connection is open, or the duration of the audio you send if that is longer, rounded up to the nearest second
Expand Down Expand Up @@ -100,7 +100,7 @@

When you are at the limit, the [native API](/api-reference/introduction),
including `POST /v1/asr`, returns 429 without a `Retry-After` header. Retry
with exponential backoff.

Check warning on line 103 in developer-guide/models-pricing/pricing-and-rate-limits.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/pricing-and-rate-limits.mdx#L103

Did you really mean 'backoff'?

### Convert Concurrency to QPS or QPM

Expand Down Expand Up @@ -148,7 +148,7 @@
| AI companion short reply | 2 seconds | ~150 QPM | ~450 QPM | ~1,500 QPM |
| Voice-agent support turn | 4 seconds | ~75 QPM | ~225 QPM | ~750 QPM |
| Narration paragraph | 12 seconds | ~25 QPM | ~75 QPM | ~250 QPM |
| Audiobook or long-form synthesis | 60 seconds | ~5 QPM | ~15 QPM | ~50 QPM |

Check warning on line 151 in developer-guide/models-pricing/pricing-and-rate-limits.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/models-pricing/pricing-and-rate-limits.mdx#L151

Did you really mean 'Audiobook'?

Treat these numbers as examples, not guarantees. Your actual QPS and QPM depend
on text length, model, reference audio, output format, network path, streaming
Expand Down
3 changes: 3 additions & 0 deletions developer-guide/sdk-guide/cookbook/realtime-transcription.mdx
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
---
title: "Transcribe a live audio stream in realtime"

Check warning on line 2 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L2

Did you really mean 'realtime'?
description: "Stream audio to transcribe-stream-1 over a WebSocket and print drafts, turn events and final transcripts as they arrive"
icon: "tower-broadcast"
tag: "Preview"
---

import Prerequisites from "/snippets/prerequisites.mdx";

<Note>`transcribe-stream-1` is in preview.</Note>

## Prerequisites

<Prerequisites />

The SDKs do not wrap the realtime speech-to-text endpoint yet, so this recipe speaks the WebSocket protocol directly. Install a WebSocket client and a MessagePack library:

Check warning on line 16 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L16

Did you really mean 'SDKs'?

Check warning on line 16 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L16

Did you really mean 'realtime'?

<CodeGroup>

Expand Down Expand Up @@ -233,7 +236,7 @@

</CodeGroup>

The audio goes in `data` as the raw PCM bytes, which must travel as MessagePack `bin`, not as base64 text: msgpack for Python packs `bytes` as `bin` by default (do not set `use_bin_type=False`), and `@msgpack/msgpack` packs a `Buffer` or `Uint8Array` as `bin`.

Check warning on line 239 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L239

Did you really mean 'msgpack'?

The server closes a session after 30 seconds without any client message, but a WebSocket ping counts as one, and Python's `websockets` sends a ping every 20 seconds by default, so the session stays open even if your audio stops; see [Capture audio well](/developer-guide/sdk-guide/cookbook/realtime-voice-agent#capture-audio-well) for a live source that can stall.

Expand Down Expand Up @@ -487,10 +490,10 @@

Install `aiohttp`, put both files in one folder, run `python relay.py`, and open `http://localhost:8080`.

- The page captures audio at the device's own sample rate and sends that rate in `config`; the server converts it, so the page does not resample. The rate must be one of the supported rates.

Check warning on line 493 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L493

Did you really mean 'resample'?
- An `AudioWorklet` converts the samples to 16-bit PCM and posts about 100 ms per message.
- **Stop** sends `audio_done`, so the server commits the last turn before it closes the connection.
- The relay forwards the API's close code to the page. In production, add your own user authentication, per-user session limits and input checks to the relay.

Check warning on line 496 in developer-guide/sdk-guide/cookbook/realtime-transcription.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-transcription.mdx#L496

Did you really mean 'API's'?

## Related

Expand Down
3 changes: 3 additions & 0 deletions developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
---
title: "Build a realtime voice agent: streaming speech in, reply, speech out"

Check warning on line 2 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L2

Did you really mean 'realtime'?
description: "Use realtime speech to text to know when the user has finished, start replies early, keep talking through backchannels and stop on interruptions"

Check warning on line 3 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L3

Did you really mean 'realtime'?

Check warning on line 3 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L3

Did you really mean 'backchannels'?
icon: "headset"
tag: "Preview"
---

import Prerequisites from "/snippets/prerequisites.mdx";

<Note>
This guide is for building your own voice agent on the realtime speech-to-text

Check warning on line 11 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L11

Did you really mean 'realtime'?
API. To have Fish Audio run the whole agent for you, see
[Agents](/agents/overview).
</Note>

<Note>`transcribe-stream-1` is in preview.</Note>

## Prerequisites

<Prerequisites />
Expand All @@ -22,7 +25,7 @@
pip install "websockets>=14" msgpack
```

Read [Transcribe a live audio stream](/developer-guide/sdk-guide/cookbook/realtime-transcription) first if you have not used the realtime endpoint before.

Check warning on line 28 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L28

Did you really mean 'realtime'?

## How a conversation flows

Expand Down Expand Up @@ -257,7 +260,7 @@

Without `agent_speaking`, every acknowledgement takes the turn, and you have to tell "mm-hm" from an interruption yourself.

## Handle interruptions and backchannels

Check warning on line 263 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L263

Did you really mean 'backchannels'?

While your agent is speaking:

Expand All @@ -282,7 +285,7 @@
{
"type": "config",
"audio": { "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 },
"turn_detection": { "mode": "auto", "eager": true },

Check warning on line 288 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L288

Did you really mean 'turn_detection'?
"context": {
"history": [
{
Expand All @@ -291,7 +294,7 @@
},
{ "role": "user", "text": "Hi, I'm calling about my prescription." }
],
"terms": ["Main Street Pharmacy", "Lisinopril"]

Check warning on line 297 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L297

Did you really mean 'Lisinopril'?
}
}
```
Expand All @@ -301,13 +304,13 @@
- **Keep echo cancellation on.** Without it, the agent's own voice reaches the microphone, opens turns and can be taken for the user. In a browser, request `echoCancellation: true` from `getUserMedia`. Some echo can remain even with echo cancellation. It is usually committed as a non-user turn, but it still opens turns (`speech_start`) while your agent speaks, which is why ducks need a cap. If the start of most replies is ducked, echo is leaking: check echo cancellation, or test with headphones.
- **Turn automatic gain control off where you can** (`autoGainControl: false` in a browser), so quiet background sound is not amplified between phrases.
- **Test with headphones first.** They keep the agent's voice out of the microphone, so you can check turn-taking before you tune echo cancellation.
- **Send audio as you capture it**, mono 16-bit PCM in 80–100 ms messages, at the device's own rate if it is one of the supported rates. The server converts the rate, so do not resample. Telephony audio at 8 kHz is supported; decode μ-law or A-law call audio to 16-bit PCM first.

Check warning on line 307 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L307

Did you really mean 'resample'?
- **Watch your audio source.** The server closes a session after 30 seconds without any client message (`idle_timeout`, close 4408), but a WebSocket ping counts as one, and Python's `websockets` sends a ping every 20 seconds by default. With that default, a session stays open even when your audio stops, and it counts against your concurrency limit until you close it or the 60-minute session limit ends it. If your audio source can stall, watch it in your own code and send `audio_done` or close the connection when it stops, or pass `ping_interval=None` to `websockets.connect` to leave idle detection to the server. Node's `ws` sends no pings by default.
- **Keep keys on your server.** In a browser, capture audio and relay it through your server, as in [Capture from a browser microphone](/developer-guide/sdk-guide/cookbook/realtime-transcription#capture-from-a-browser-microphone). Send `agent_speaking` on that same connection so it stays in order with the audio.

## Languages

Turn detection, including eager end of turn, backchannel detection and background awareness, works on speech in every language. In some languages, a turn can end a little later after the speaker stops. For faster turns, use `vad` mode (500 ms or more is a good start for `silence_timeout_ms`), or end turns yourself with `finalize`.

Check warning on line 313 in developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

developer-guide/sdk-guide/cookbook/realtime-voice-agent.mdx#L313

Did you really mean 'backchannel'?

## Connect your LLM and speech

Expand Down
5 changes: 4 additions & 1 deletion features/speech-to-text/realtime.mdx
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
---
title: "Realtime Speech to Text"

Check warning on line 2 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L2

Did you really mean 'Realtime'?
sidebarTitle: "Realtime"
description: "Transcribe live audio turn by turn with transcribe-stream-1: turn detection that knows when the user has finished, backchannel and background awareness, and conversation context"

Check warning on line 4 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L4

Did you really mean 'backchannel'?
icon: "microphone-lines"
tag: "Preview"
---

Stream live audio over a WebSocket and get the transcript turn by turn while people speak. `transcribe-stream-1` does more than transcribe: it decides when the user has finished their turn, usually tells the user talking to your app apart from background sound, and notices when a "mm-hm" is only an acknowledgement. This helps your voice agent decide when to answer, when to keep talking and when to stop. Connect to `wss://api.fish.audio/v1/asr/realtime`, send audio as you capture it, and read the events the server sends back.

<Note>`transcribe-stream-1` is in preview.</Note>

<CardGroup cols={3}>
<Card
title="API reference"
Expand All @@ -27,7 +30,7 @@
icon="headset"
href="/developer-guide/sdk-guide/cookbook/realtime-voice-agent"
>
Turn-taking, interruptions and backchannels.

Check warning on line 33 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L33

Did you really mean 'backchannels'?
</Card>
</CardGroup>

Expand Down Expand Up @@ -61,19 +64,19 @@

To transcribe a recording you already have, use [`POST /v1/asr`](/features/speech-to-text) with `transcribe-1-pro` instead:

| | Realtime (`/v1/asr/realtime`) | File (`POST /v1/asr`) |

Check warning on line 67 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L67

Did you really mean 'Realtime'?
| -------------- | -------------------------------------------------------------- | ---------------------------------------------------------- |
| Model | `transcribe-stream-1` | `transcribe-1-pro` |
| Model | `transcribe-stream-1` (Preview) | `transcribe-1-pro` |
| Input | Live 16-bit PCM over a WebSocket | A complete audio file (WAV, MP3, Opus, and more) |
| Output | Events per turn: drafts, turn signals and the final transcript | One transcript, with optional timestamps and speaker turns |
| Turn detection | Yes | No |
| Session length | Up to 60 minutes | Recordings up to 60 minutes |

The realtime endpoints of the [compatible APIs](/developer-guide/compat/capabilities) are separate from this endpoint. They use manual commit and have no server-side turn detection.

Check warning on line 75 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L75

Did you really mean 'realtime'?

## Quick start

The SDKs do not wrap this endpoint; you connect with any WebSocket client and exchange MessagePack messages.

Check warning on line 79 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L79

Did you really mean 'SDKs'?

<Steps>
<Step title="Connect">
Expand Down Expand Up @@ -102,7 +105,7 @@
{
"type": "config",
"audio": { "format": "pcm_s16le", "sample_rate": 16000, "channels": 1 },
"turn_detection": { "mode": "auto", "eager": true },

Check warning on line 108 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L108

Did you really mean 'turn_detection'?
"results": { "drafts": true }
}
```
Expand Down Expand Up @@ -140,7 +143,7 @@

With `turn_detection.eager: true`, you receive `eager_end_of_turn` as soon as the turn has probably ended, before the final transcript. Its text is exactly the text `committed` will have, unless `turn_resumed` follows because the user went on. A voice agent can start its LLM on the eager text and save that time.

### Backchannel detection

Check warning on line 146 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L146

Did you really mean 'Backchannel'?

Tell the server when your agent is speaking with `agent_speaking`. While it speaks, a short "mm-hm", "yeah", "right" or "uh-huh" from the user produces `user_backchannel`: the agent keeps talking. A real interruption produces `user_turn_taken`: the agent stops.

Expand All @@ -166,14 +169,14 @@
| Mode | A turn ends when | Use it for |
| ---------------- | ------------------------------------------------------------------ | -------------------------------------- |
| `auto` (default) | Turn detection judges that the user has finished. | Voice agents and conversations. |
| `vad` | A fixed silence has passed (`silence_timeout_ms`, default 700 ms). | Simple silence-based endpointing. |

Check warning on line 172 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L172

Did you really mean 'endpointing'?
| `manual` | You send `finalize`. | Push-to-talk, or your own endpointing. |

Check warning on line 173 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L173

Did you really mean 'endpointing'?

Eager end of turn, backchannel detection and background awareness are available in `auto` mode, when the session starts in `auto` (the default). In `vad` and `manual` mode, and after a later switch to `auto` from another mode, every turn is a user turn. See [Turn detection modes](/api-reference/endpoint/websocket/asr-realtime#turn-detection-modes) for details.

Check warning on line 175 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L175

Did you really mean 'backchannel'?

## Languages

Smart Turn Detection, including eager end of turn, backchannel detection and background awareness, works on speech in every language. In some languages, a turn can end a little later after the speaker stops.

Check warning on line 179 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L179

Did you really mean 'backchannel'?

`draft`, `eager_end_of_turn` and `committed` carry the language of their own text as `language_code` when it can be identified, one of 29 [supported codes](/api-reference/endpoint/websocket/asr-realtime#language).

Expand All @@ -191,7 +194,7 @@

### Audio

Send 16-bit little-endian PCM, mono, at 8000, 12000, 16000, 22050, 24000, 32000, 44100 or 48000 Hz. The server converts the sample rate, so you never need to resample. Compressed formats such as Opus or MP3 are not accepted on this endpoint.

Check warning on line 197 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L197

Did you really mean 'resample'?

### Speakers

Expand All @@ -207,7 +210,7 @@

### Pricing

`transcribe-stream-1` costs $0.36 per hour, billed per second of session duration as its own line item, Realtime ASR. See [ASR pricing](/developer-guide/models-pricing/pricing-and-rate-limits#automatic-speech-recognition-asr-models).

Check warning on line 213 in features/speech-to-text/realtime.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

features/speech-to-text/realtime.mdx#L213

Did you really mean 'Realtime'?

## Going further

Expand Down
10 changes: 5 additions & 5 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

- [OpenAPI](https://docs.fish.audio/api-reference/openapi.json): REST API schema for text-to-speech, speech-to-text, wallet, and voice model endpoints.
- [AsyncAPI](https://docs.fish.audio/api-reference/asyncapi.yml): WebSocket schema for real-time text-to-speech streaming.
- [AsyncAPI: Realtime Speech to Text](https://docs.fish.audio/api-reference/asyncapi-asr-realtime.yml): WebSocket schema for realtime speech-to-text (transcribe-stream-1).
- [AsyncAPI: Realtime Speech to Text](https://docs.fish.audio/api-reference/asyncapi-asr-realtime.yml): WebSocket schema for realtime speech-to-text (transcribe-stream-1, Preview).

## Core REST API

Expand All @@ -28,7 +28,7 @@
- [Update Model](https://docs.fish.audio/api-reference/endpoint/model/update-model.md): Update an existing model.
- [Delete Model](https://docs.fish.audio/api-reference/endpoint/model/delete-model.md): Delete an existing model.
- [WebSocket TTS Streaming](https://docs.fish.audio/api-reference/endpoint/websocket/tts-live.md): Real-time text-to-speech streaming via WebSocket.
- [WebSocket Realtime Speech to Text](https://docs.fish.audio/api-reference/endpoint/websocket/asr-realtime.md): Stream audio to wss://api.fish.audio/v1/asr/realtime (transcribe-stream-1) and receive transcripts per turn; every client and server message, turn detection modes, errors, close codes and limits.
- [WebSocket Realtime Speech to Text](https://docs.fish.audio/api-reference/endpoint/websocket/asr-realtime.md): Stream audio to wss://api.fish.audio/v1/asr/realtime (transcribe-stream-1, Preview) and receive transcripts per turn; every client and server message, turn detection modes, errors, close codes and limits.

## SDKs

Expand All @@ -48,9 +48,9 @@
- [Text to Speech Guide](https://docs.fish.audio/developer-guide/core-features/text-to-speech.md): Convert text to natural-sounding speech with Fish Audio.
- [Drama 3](https://docs.fish.audio/features/text-to-speech/drama-3.md): Text-to-speech model (drama-3) directed with plain-language directions in square brackets, sounds and pauses, paired tags for specific words, and multi-speaker dialogue.
- [Speech to Text Guide](https://docs.fish.audio/features/speech-to-text.md): Transcribe audio with transcribe-1-pro, including multi-speaker conversations with speaker turns, long recordings and inline emotion cues.
- [Realtime Speech to Text](https://docs.fish.audio/features/speech-to-text/realtime.md): Transcribe live audio turn by turn with transcribe-stream-1: turn detection, eager end of turn, backchannel detection and conversation context for voice agents.
- [Transcribe a Live Audio Stream](https://docs.fish.audio/developer-guide/sdk-guide/cookbook/realtime-transcription.md): Python and Node.js clients for the realtime speech-to-text WebSocket, plus a browser microphone relay.
- [Build a Realtime Voice Agent](https://docs.fish.audio/developer-guide/sdk-guide/cookbook/realtime-voice-agent.md): Agent loop on realtime speech to text: eager replies, agent_speaking, barge-in and backchannels.
- [Realtime Speech to Text](https://docs.fish.audio/features/speech-to-text/realtime.md): Transcribe live audio turn by turn with transcribe-stream-1 (Preview): turn detection, eager end of turn, backchannel detection and conversation context for voice agents.
- [Transcribe a Live Audio Stream](https://docs.fish.audio/developer-guide/sdk-guide/cookbook/realtime-transcription.md): Python and Node.js clients for the realtime speech-to-text WebSocket (transcribe-stream-1, Preview), plus a browser microphone relay.
- [Build a Realtime Voice Agent](https://docs.fish.audio/developer-guide/sdk-guide/cookbook/realtime-voice-agent.md): Agent loop on realtime speech to text (transcribe-stream-1, Preview): eager replies, agent_speaking, barge-in and backchannels.
- [Voice Design Guide](https://docs.fish.audio/features/voice-design.md): Generate candidate voices from natural-language prompts.
- [Creating Voice Models](https://docs.fish.audio/developer-guide/core-features/creating-models.md): Learn how to create custom voice models with Fish Audio.
- [Emotion Control](https://docs.fish.audio/developer-guide/core-features/emotions.md): Add natural emotions and expressions to your AI-generated speech.
Expand Down
2 changes: 1 addition & 1 deletion overview/capabilities.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
</Card>

<Card title="Story Studio" icon="book-open" href="/overview/platform">
Produce multi-speaker, long-form audio: audiobooks and narration.

Check warning on line 52 in overview/capabilities.mdx

View check run for this annotation

Mintlify / Mintlify Validation (hanabiaiinc) - vale-spellcheck

overview/capabilities.mdx#L52

Did you really mean 'audiobooks'?
</Card>

<Card title="Music & Sound Effects" icon="music" href="/overview/platform">
Expand All @@ -75,7 +75,7 @@
- **`transcribe-1-pro`**: the recommended model, for everything from short clips to multi-speaker conversations (speaker markers and speaker turns) and long recordings, with inline emotion cues. Send `model: transcribe-1-pro` explicitly.
- **`transcribe-1`**: general transcription of short recordings. A request whose `model` header is missing or not an exact match is served by this model.

To transcribe live audio, `transcribe-stream-1` streams transcripts turn by turn over a WebSocket, with turn detection for voice agents. It is available through the API; see [Realtime Speech to Text](/features/speech-to-text/realtime).
To transcribe live audio, `transcribe-stream-1` (Preview) streams transcripts turn by turn over a WebSocket, with turn detection for voice agents. It is available through the API; see [Realtime Speech to Text](/features/speech-to-text/realtime).

See [Models Overview](/developer-guide/models-pricing/models-overview) and [Choosing a Model](/developer-guide/models-pricing/choosing-a-model) for the full lineup and languages, and [Speech to Text](/features/speech-to-text#input-audio) for transcription formats and limits.

Expand Down
Loading