Universal.Operative.Sdk.Realtime.OpenAI 1.2.0

Universal.Operative.Sdk.Realtime.OpenAI

About

Universal.Operative.Sdk.Realtime.OpenAI adapts the OpenAI Realtime API to Universal.Operative.Sdk.Realtime's provider-neutral IModel/IRealtimeSession/SessionEvent model. Ported from the Ong.Bot/Operative.Marin prototype.

Two connection paths in this release: Model.ConnectAsync (server-mediated WebSocket — audio relays through your process) and Model.ConnectForCallAsync (SIP passthrough — audio flows directly between the caller and OpenAI, never touching your process). Browser-direct WebRTC (an ephemeral-key handshake so a browser tab talks to OpenAI without your server in the audio path) isn't implemented yet.

How to Use

Installation

dotnet add package Universal.Operative.Sdk
dotnet add package Universal.Operative.Sdk.Realtime
dotnet add package Universal.Operative.Sdk.Realtime.OpenAI

Quick start

using Universal.Operative.Sdk.Realtime;
using Universal.Operative.Sdk.Realtime.OpenAI;

var model = new Model(apiKey, modelId: "gpt-realtime", defaultVoice: "marin");

IRealtimeSession session = await model.ConnectAsync(new RealtimeSessionOptions
{
    SystemPrompt = [new TextBlock("You are a helpful voice assistant.")],
    Tools = [myTool],
});

IOperative operative = OperativeBuilder.Create(session).AddTool(myTool).Build();

await foreach (SessionOutputEvent e in operative.RunAsync())
{
    switch (e)
    {
        case SessionReadyEvent: Console.WriteLine("connected"); break;
        case TranscriptDeltaEvent t: Console.WriteLine($"{t.Role}: {t.Text}"); break;
        case AudioOutputDeltaEvent a: /* play a.Audio (24kHz PCM16) */ break;
        case InterruptedEvent: /* stop playback immediately */ break;
    }
}

await operative.SendAsync(new AppendAudioInputEvent { Audio = micChunk });

Fidelity notes

• Capabilities: AudioInProcess | ToolCalling | Transcripts — audio flows through this same WebSocket, and there's no native session resumption (SessionCapabilities.NativeResumption/ClientAssistedResumption are both unset).
• Tool results: the full ToolResult content (all blocks, not just text) is flattened into OpenAI's single-string function_call_output — text blocks contribute their text verbatim, anything else is JSON-serialized rather than silently dropped.
• Tool choice: RealtimeSessionOptions.ToolChoice is sent as OpenAI's own tool_choice at connect time (same shape as Chat Completions/Responses — a bare string for auto/required/none, a structured object to force one specific tool).
• Raw passthrough: OpenAI's Realtime API has substantially more event types than are mapped to canonical events here (item lifecycle events, rate-limit updates, built-in MCP tool events, text-modality deltas). Anything not explicitly mapped arrives as a RawPassthroughEvent rather than being dropped.
• Mid-session reconfiguration and other provider-specific sends: OpenAI's session.update (changing instructions/voice/tools/tool-choice after connecting) has no canonical SessionInputEvent yet — unconfirmed whether Gemini or Nova Sonic support anything equivalent, so it isn't promoted. RawSendEvent covers this and more generally: it resolves its RawJson against every client event this adapter can send (session.update, response.create, response.cancel, the conversation.item.*/input_audio_buffer.* family, etc.) by matching its "type" field. A constructed event (e.g. SessionUpdateEvent from Universal.OpenAI.Realtime.Client) as Payload also works, and is used as-is with no deserialization. Explicitly requesting a response with no new input first (needed because OpenAI's server-side turn detection doesn't always resume on its own after interrupt_response auto-cancels a response mid-generation) goes through this same path: RawSendEvent { RawJson = "{\"type\":\"response.create\"}" }. Any input event this doesn't recognize still throws NotSupportedException.