Skip to content

Sample_VoiceAgent.md

Latest commit

 

History

History
497 lines (446 loc) · 22.8 KB

Sample_VoiceAgent.md

File metadata and controls

497 lines (446 loc) · 22.8 KB
Raw

Use a Foundry voice agent with REST and realtime APIs

This sample uses the published Azure.AI.Projects and Azure.AI.Projects.Agents 3.0.0-beta.3 packages to configure and manage a voice agent over REST, exchange OpenAI Realtime GA events over its Foundry WebSocket endpoint, use an Azure neural voice with a runtime pitch update, stream PCM audio, and read persisted conversations and recordings.

See README.md for prerequisites, the NuGet SDK packages, environment variables, and commands to build and run the standalone console application. Both temporary-agent and existing-agent sessions persist transcripts and audio with Store = true. The temporary agent is deleted when the run finishes.

Set FOUNDRY_VOICE_AGENT_NAME to run against an existing voice agent. In that mode, the sample uses the agent's latest version and does not create, disable, enable, or delete the agent.

When creating a temporary agent, set FOUNDRY_VOICE_MODEL_TYPE to managed for a service-managed model such as gpt-realtime. Set FOUNDRY_VOICE_MODEL_NAME to select a voice model independently of the text model used by other samples; otherwise, the sample uses FOUNDRY_MODEL_NAME. If the model type is omitted, the selected name is treated as a self-deployed Foundry model deployment.

The temporary agent uses en-US-AvaNeural with azure-standard speech synthesis and 24 kHz PCM16 mono audio. Set FOUNDRY_VOICE_INPUT_AUDIO_PATH to supply input in that format. When it is omitted, the temporary-agent sample streams its first spoken response back as input, so the audio-streaming path still runs without a microphone or an audio file. Set FOUNDRY_VOICE_OUTPUT_AUDIO_PATH to choose the output file; otherwise, a uniquely named file is written to the temporary directory.

Create an AIProjectClient with your project endpoint and credential, then use its AgentAdministrationClient property for agent management. The executable sample suppresses AAIP001, AAIP002, and OPENAI002 for these preview APIs.

  1. Create a managed or self-deployed voice agent. Set Store to true when its conversations and audio should be available through the Foundry REST APIs.
VoiceAgentDefinition definition = new()
{
    ModelType = voiceModelType,
    Model = modelDeploymentName,
    Instructions = "Respond briefly and helpfully.",
    Audio = new VoiceAgentAudioConfig
    {
        Input = new VoiceAgentAudioInputConfig
        {
            Format = CreatePcmAudioFormat(24000),
            NoiseReduction = new VoiceAgentNoiseReduction(VoiceAgentNoiseReductionType.NearField),
            TurnDetection = new VoiceAgentServerVadTurnDetection
            {
                Threshold = 0.5,
                PrefixPaddingMs = 300,
                SilenceDurationMs = 500
            },
            Transcription = new VoiceAgentInputTranscription(VoiceAgentInputTranscriptionModel.Whisper1)
        },
        Output = new VoiceAgentAudioOutputConfig
        {
            Format = CreatePcmAudioFormat(24000),
            Voice = "en-US-AvaNeural",
            VoiceType = VoiceType.AzureStandard
        }
    },
    Store = true
};
definition.OutputModalities.Add(VoiceOutputModality.Audio);
definition.Tools.Add(new VoiceAgentEndConversationSystemTool());
ClientResult<ProjectsAgentVersion> createResult = await agentsClient.CreateAgentVersionAsync(
    agentName,
    new ProjectsAgentVersionCreationOptions(definition),
    foundryFeatures: "VoiceAgents=V1Preview");
agentVersion = createResult;
  1. Retrieve the agent and its versions, and use the same enable and disable operations shared by other Foundry agent kinds.
ProjectsAgentRecord agent = await agentsClient.GetAgentAsync(agentName);
ProjectsAgentVersion version = await agentsClient.GetAgentVersionAsync(
    agentName,
    agentVersion.Version);
Console.WriteLine($"Voice agent {agent.Name}, version {version.Version}");

await foreach (ProjectsAgentVersion listedVersion in agentsClient.GetAgentVersionsAsync(agentName))
{
    Console.WriteLine($"Version {listedVersion.Version}: {listedVersion.Description}");
}

ClientResult disableResult = await agentsClient.DisableAgentAsync(agentName);
ClientResult enableResult = await agentsClient.EnableAgentAsync(agentName);
  1. Connect through AIProjectClient.ProjectsRealtimeClient, add a text turn, and stream the PCM16 audio response. RealtimeSessionClientOptions.QueryString selects the agent version and enables persistence. The ReceiveUpdatesAsync(RequestOptions) protocol overload exposes raw events through GetRawResponse().Content, preserving Foundry-specific fields without reserializing OpenAI models.
ProjectsRealtimeClient realtimeClient = projectClient.ProjectsRealtimeClient;
using CancellationTokenSource timeout = new(TimeSpan.FromMinutes(5));
BetaVoiceAgentsConversations conversationsClient = agentsClient.GetBetaVoiceAgentEndpointConversations();
HashSet<string> newConversationIds = new();
bool hasGreeting = agentVersion.Definition is VoiceAgentDefinition voiceDefinition
    && voiceDefinition.Greeting is not null;
RealtimeSessionClientOptions connectionOptions = new()
{
    QueryString = $"agent_version={Uri.EscapeDataString(agentVersion.Version)}&store=true"
};

using RealtimeSessionClient session = await realtimeClient.StartSessionAsync(
    agentName,
    intent: null,
    options: connectionOptions,
    cancellationToken: timeout.Token);
newConversationIds.Add(await WaitForSessionReadyAsync(session, hasGreeting, timeout.Token));

await session.AddItemAsync(
    RealtimeItem.CreateUserMessageItem("Say hello in one sentence."),
    cancellationToken: timeout.Token);
await session.StartResponseAsync(cancellationToken: timeout.Token);

using MemoryStream responseAudio = new();
bool responseCompleted = false;
await foreach (ClientResult update in ReceiveSessionUpdatesAsync(session, timeout.Token))
{
    using JsonDocument document = JsonDocument.Parse(update.GetRawResponse().Content);
    RealtimeServerUpdateKind eventType = GetEventType(document.RootElement);
    LogRealtimeEvent(eventType, document.RootElement);
    if (eventType == RealtimeServerUpdateKind.ResponseOutputAudioDelta)
    {
        byte[] audioChunk = Convert.FromBase64String(document.RootElement.GetProperty("delta").GetString()
            ?? throw new InvalidOperationException("The audio delta was null."));
        await responseAudio.WriteAsync(audioChunk, 0, audioChunk.Length, timeout.Token);
    }
    else if (eventType == RealtimeServerUpdateKind.ResponseDone)
    {
        responseCompleted = !IsCancelledResponse(document.RootElement);
        break;
    }
}
if (!responseCompleted)
{
    throw new InvalidOperationException("The voice session closed or its response was cancelled before completion.");
}
if (responseAudio.Length == 0)
{
    throw new InvalidOperationException("The voice response did not contain audio.");
}
Console.WriteLine($"Received {responseAudio.Length} bytes of PCM response audio.");

WaitForSessionReadyAsync captures the canonical persisted conversation ID from session.created and drains any configured greeting before the first user turn. The executable sample source includes this helper and the audio-streaming and persistence helpers below. Service errors, unsuccessful responses, and premature session closure fail the sample rather than producing a successful-looking empty result.

  1. Change the Azure voice pitch for the active session. This does not change the stored agent definition. The sample only applies this update to the temporary Azure-voice agent it created.
await session.SendCommandAsync(BinaryData.FromObjectAsJson(new
{
    type = "session.update",
    session = new
    {
        type = "realtime",
        audio = new
        {
            output = new
            {
                voice = "en-US-AvaNeural",
                voice_type = "azure-standard",
                pitch = "+10%"
            }
        }
    }
}), new RequestOptions { CancellationToken = timeout.Token });
  1. Send a follow-up turn that asks the agent to end the conversation, which exercises the end_conversation system tool registered on the agent definition end-to-end.
// The agent was configured with the "end_conversation" system tool. Prompting the model
// to end the conversation exercises that tool end-to-end: the service invokes it as a
// function_call item and reflects the outcome in the response, without the client having
// to submit a function_call_output (system tools are handled entirely server-side, unlike
// custom/user-defined function tools).
await session.AddItemAsync(
    RealtimeItem.CreateUserMessageItem("Please say a brief goodbye and then end our conversation."),
    cancellationToken: timeout.Token);
await session.StartResponseAsync(cancellationToken: timeout.Token);

responseCompleted = false;
await foreach (ClientResult update in ReceiveSessionUpdatesAsync(session, timeout.Token))
{
    using JsonDocument document = JsonDocument.Parse(update.GetRawResponse().Content);
    RealtimeServerUpdateKind eventType = GetEventType(document.RootElement);
    LogRealtimeEvent(eventType, document.RootElement);
    if (eventType == RealtimeServerUpdateKind.ResponseDone)
    {
        responseCompleted = !IsCancelledResponse(document.RootElement);
        break;
    }
}
if (!responseCompleted)
{
    throw new InvalidOperationException("The voice session closed or its goodbye response was cancelled before completion.");
}
  1. Stream an audio turn. A newly created agent's first spoken response is used as input by default; set FOUNDRY_VOICE_INPUT_AUDIO_PATH to use a raw PCM16, mono, 24 kHz input file instead. Set FOUNDRY_VOICE_OUTPUT_AUDIO_PATH to choose where the streamed PCM16 response is written; otherwise the sample uses a uniquely named file in the system temporary directory.
if (!string.IsNullOrEmpty(inputAudioPath) || deleteAgent)
{
    using RealtimeSessionClient audioSession = await realtimeClient.StartSessionAsync(
        agentName,
        intent: null,
        options: connectionOptions,
        cancellationToken: timeout.Token);
    newConversationIds.Add(await WaitForSessionReadyAsync(audioSession, hasGreeting, timeout.Token));
    using Stream inputPcm = string.IsNullOrEmpty(inputAudioPath)
        ? new MemoryStream(responseAudio.ToArray())
        : File.OpenRead(inputAudioPath);
    using FileStream outputPcm = File.Create(outputAudioPath);

    await StreamAudioTurnAsync(
        audioSession,
        inputPcm,
        outputPcm,
        appendTrailingSilence: true,
        cancellationToken: timeout.Token);
    await CloseSessionAsync(audioSession, timeout.Token);
    if (outputPcm.Length == 0)
    {
        throw new InvalidOperationException("The streaming response did not contain audio.");
    }
    Console.WriteLine($"Streamed response audio to {outputAudioPath}");
}
  1. The streaming helper sends input and receives output concurrently. It accepts any Stream, so applications can replace the file streams with microphone and speaker adapters without adding an audio-device dependency to the SDK.
public static async Task<string?> StreamAudioTurnAsync(
    RealtimeSessionClient session,
    Stream inputPcm,
    Stream outputPcm,
    bool appendTrailingSilence = false,
    CancellationToken cancellationToken = default)
{
    Task<string?> receiveTask = ReceiveOutputAsync();
    Task sendTask = SendInputAsync();
    await Task.WhenAll(sendTask, receiveTask);
    return await receiveTask;

    async Task SendInputAsync()
    {
        const int bytesPerSecond = 24000 * sizeof(short);
        const int chunkSize = bytesPerSecond / 20;
        byte[] buffer = new byte[chunkSize];

        while (true)
        {
            int bytesRead = await inputPcm.ReadAsync(buffer, 0, buffer.Length, cancellationToken);
            if (bytesRead == 0)
            {
                break;
            }

            await session.SendInputAudioAsync(
                BinaryData.FromBytes(buffer.AsMemory(0, bytesRead)),
                cancellationToken);
            await Task.Delay(TimeSpan.FromSeconds((double)bytesRead / bytesPerSecond), cancellationToken);
        }

        if (appendTrailingSilence)
        {
            Array.Clear(buffer, 0, buffer.Length);
            for (int i = 0; i < 20; i++)
            {
                await session.SendInputAudioAsync(BinaryData.FromBytes(buffer), cancellationToken);
                await Task.Delay(TimeSpan.FromMilliseconds(50), cancellationToken);
            }
        }
    }

    async Task<string?> ReceiveOutputAsync()
    {
        string? conversationId = null;
        await foreach (ClientResult update in ReceiveSessionUpdatesAsync(session, cancellationToken))
        {
            using JsonDocument document = JsonDocument.Parse(update.GetRawResponse().Content);
            RealtimeServerUpdateKind eventType = GetEventType(document.RootElement);
            LogRealtimeEvent(eventType, document.RootElement);
            conversationId = GetConversationId(document.RootElement) ?? conversationId;
            if (eventType == RealtimeServerUpdateKind.ResponseOutputAudioDelta)
            {
                byte[] audioChunk = Convert.FromBase64String(document.RootElement.GetProperty("delta").GetString()
                    ?? throw new InvalidOperationException("The audio delta was null."));
                await outputPcm.WriteAsync(audioChunk, 0, audioChunk.Length, cancellationToken);
            }
            else if (eventType == RealtimeServerUpdateKind.ResponseDone)
            {
                if (IsCancelledResponse(document.RootElement))
                {
                    continue;
                }
                return conversationId;
            }
        }
        throw new InvalidOperationException("The voice session closed before the audio response completed.");
    }
}

The executable sample's ReceiveSessionUpdatesAsync helper uses the SDK's protocol overload and handles empty WebSocket close frames so premature closure still reports a sample error. CloseSessionAsync closes the SDK session's WebSocket before reading persisted data. GetEventType reads the raw event's type into RealtimeServerUpdateKind; the other event helpers retain Foundry conversation IDs and report service errors.

  1. Use BetaVoiceAgentsConversations to retrieve conversations persisted by an agent configured with Store = true.
foreach (string conversationId in newConversationIds)
{
    await WaitForConversationPersistenceAsync(
        conversationsClient,
        agentName,
        conversationId,
        timeout.Token);
    string assistantItemId = await ReadPersistedConversationAsync(
        conversationsClient,
        agentName,
        conversationId,
        timeout.Token);

    using MemoryStream conversationAudio = new();
    await DownloadConversationAudioAsync(
        conversationsClient,
        agentName,
        conversationId,
        conversationAudio,
        timeout.Token);
    if (conversationAudio.Length > 0)
    {
        Console.WriteLine($"Downloaded {conversationAudio.Length} bytes of conversation audio.");
    }

    if (!string.IsNullOrEmpty(assistantItemId))
    {
        using MemoryStream itemAudio = new();
        await DownloadConversationItemAudioAsync(
            conversationsClient,
            agentName,
            conversationId,
            assistantItemId,
            itemAudio,
            timeout.Token);
        if (itemAudio.Length > 0)
        {
            Console.WriteLine($"Downloaded {itemAudio.Length} bytes of assistant item audio.");
        }
    }
}
  1. Read a persisted conversation, its inference responses, and the ordered response and conversation items.
private static async Task<string> ReadPersistedConversationAsync(
    BetaVoiceAgentsConversations conversationsClient,
    string agentName,
    string conversationId,
    CancellationToken cancellationToken = default)
{
    string? assistantItemId = null;
    ClientResult<VoiceConversation> conversationResult = await conversationsClient.GetAgentConversationAsync(
        agentName,
        conversationId,
        cancellationToken);
    VoiceConversation conversation = conversationResult;
    Console.WriteLine($"Created at {conversation.CreatedOn}; status: {conversation.Status} (GetAgentConversation status: {(int)conversationResult.GetRawResponse().Status})");

    await foreach (VoiceResponse response in conversationsClient.GetAgentConversationResponsesAsync(
        agentName,
        conversationId,
        cancellationToken: cancellationToken))
    {
        string responseId = response.Id;
        ClientResult<VoiceResponse> detailResult = await conversationsClient.GetAgentConversationResponseAsync(
            agentName,
            conversationId,
            responseId,
            cancellationToken);
        VoiceResponse detail = detailResult;
        Console.WriteLine($"Response {responseId}: {detail.Status} (GetAgentConversationResponse status: {(int)detailResult.GetRawResponse().Status})");

        await foreach (RealtimeItem conversationItem in conversationsClient.GetAgentConversationResponseItemsAsync(
            agentName,
            conversationId,
            responseId,
            cancellationToken: cancellationToken))
        {
            Console.WriteLine($"Response item: {DescribeItem(conversationItem)}");
            if (assistantItemId is null
                && conversationItem is RealtimeMessageItem assistantMessage
                && assistantMessage.Role == RealtimeMessageRole.Assistant)
            {
                assistantItemId = assistantMessage.Id;
            }
        }
    }

    await foreach (RealtimeItem conversationItem in conversationsClient.GetAgentConversationItemsAsync(
        agentName,
        conversationId,
        cancellationToken: cancellationToken))
    {
        Console.WriteLine($"Conversation item: {DescribeItem(conversationItem)}");
    }
    if (assistantItemId is null)
    {
        throw new InvalidOperationException("The persisted conversation did not contain an assistant message.");
    }
    RealtimeItem persistedItem = await conversationsClient.GetAgentConversationItemAsync(
        agentName, conversationId, assistantItemId, cancellationToken);
    Console.WriteLine($"Retrieved item: {DescribeItem(persistedItem)}");
    return assistantItemId;
}

/// <summary>
/// Describes a persisted conversation item using typed pattern-matching rather than
/// round-tripping through <see cref="ModelReaderWriter.Write{T}(T, ModelReaderWriterOptions)"/>.
/// Items retrieved from the persisted-conversation-item endpoints omit large fields (e.g. inline
/// audio bytes, which are fetched separately via the dedicated audio endpoints), and re-serializing
/// such a partially-populated item can throw inside some OpenAI.Realtime content-part writers that
/// don't yet null-check every field. Typed access below sidesteps that entirely.
/// </summary>
private static string DescribeItem(RealtimeItem item) => item switch
{
    RealtimeMessageItem message => $"message (role={message.Role}, id={message.Id}, status={message.Status})",
    RealtimeFunctionCallItem functionCall => $"function_call (name={functionCall.FunctionName}, id={functionCall.Id})",
    RealtimeFunctionCallOutputItem functionCallOutput => $"function_call_output (id={functionCallOutput.Id})",
    RealtimeMcpToolCallItem mcpToolCall => $"mcp_call (name={mcpToolCall.ToolName}, id={mcpToolCall.Id})",
    RealtimeMcpToolCallApprovalRequestItem mcpApprovalRequest => $"mcp_approval_request (id={mcpApprovalRequest.Id})",
    RealtimeMcpToolCallApprovalResponseItem mcpApprovalResponse => $"mcp_approval_response (id={mcpApprovalResponse.Id})",
    RealtimeMcpToolDefinitionListItem mcpToolList => $"mcp_list_tools (id={mcpToolList.Id})",
    _ => $"unknown ({item.GetType().Name})"
};
  1. Retrieve whole-call or per-item recording metadata. Foundry-managed audio is returned as WAV content; when BlobUri is present, download the recording from the configured bring-your-own-storage account instead.
private static async Task DownloadConversationAudioAsync(
    BetaVoiceAgentsConversations conversationsClient,
    string agentName,
    string conversationId,
    Stream destination,
    CancellationToken cancellationToken = default)
{
    ClientResult<VoiceRecording> recordingResult = await conversationsClient.GetAgentConversationAudioAsync(
        agentName,
        conversationId,
        cancellationToken);
    VoiceRecording recording = recordingResult;
    Console.WriteLine($"{recording.Format}, {recording.SampleRate} Hz, {recording.Channels} channels (GetAgentConversationAudio status: {(int)recordingResult.GetRawResponse().Status})");

    if (recording.BlobUri is not null)
    {
        Console.WriteLine($"Download the bring-your-own-storage recording from {recording.BlobUri}");
        return;
    }

    ClientResult<BinaryData> contentResult = await conversationsClient.GetAgentConversationAudioContentAsync(
        agentName,
        conversationId,
        cancellationToken);
    Console.WriteLine($"GetAgentConversationAudioContent status: {(int)contentResult.GetRawResponse().Status}");
    byte[] bytes = ((BinaryData)contentResult).ToArray();
    if (bytes.Length == 0)
    {
        throw new InvalidOperationException("The conversation recording was empty.");
    }
    await destination.WriteAsync(bytes, 0, bytes.Length, cancellationToken);
}

private static async Task DownloadConversationItemAudioAsync(
    BetaVoiceAgentsConversations conversationsClient,
    string agentName,
    string conversationId,
    string itemId,
    Stream destination,
    CancellationToken cancellationToken = default)
{
    ClientResult<VoiceAudioItem> audioResult = await conversationsClient.GetAgentConversationAudioItemAsync(
        agentName,
        conversationId,
        itemId,
        cancellationToken);
    VoiceAudioItem audio = audioResult;
    Console.WriteLine($"{audio.Role}: {audio.DurationMs} (GetAgentConversationAudioItem status: {(int)audioResult.GetRawResponse().Status})");

    if (audio.BlobUri is not null)
    {
        Console.WriteLine($"Download the bring-your-own-storage item audio from {audio.BlobUri}");
        return;
    }

    ClientResult<BinaryData> contentResult = await conversationsClient.GetAgentConversationAudioItemContentAsync(
        agentName,
        conversationId,
        itemId,
        cancellationToken);
    Console.WriteLine($"GetAgentConversationAudioItemContent status: {(int)contentResult.GetRawResponse().Status}");
    byte[] bytes = ((BinaryData)contentResult).ToArray();
    if (bytes.Length == 0)
    {
        throw new InvalidOperationException("The item recording was empty.");
    }
    await destination.WriteAsync(bytes, 0, bytes.Length, cancellationToken);
}