Universal.Operative.Sdk.Realtime.OpenAI 1.0.0

About

Universal.Operative.Sdk.Realtime.OpenAI adapts the OpenAI Realtime API to Universal.Operative.Sdk.Realtime's provider-neutral IModel/IRealtimeSession/SessionEvent model. Ported from the Ong.Bot/Operative.Marin prototype.

Two connection paths in this release: Model.ConnectAsync (server-mediated WebSocket — audio relays through your process) and Model.ConnectForCallAsync (SIP passthrough — audio flows directly between the caller and OpenAI, never touching your process). Browser-direct WebRTC (an ephemeral-key handshake so a browser tab talks to OpenAI without your server in the audio path) isn't implemented yet.

How to Use

Installation

dotnet add package Universal.Operative.Sdk
dotnet add package Universal.Operative.Sdk.Realtime
dotnet add package Universal.Operative.Sdk.Realtime.OpenAI

Quick start

using Universal.Operative.Sdk.Realtime;
using Universal.Operative.Sdk.Realtime.OpenAI;

var model = new Model(apiKey, modelId: "gpt-realtime", defaultVoice: "marin");

IRealtimeSession session = await model.ConnectAsync(new RealtimeSessionOptions
{
    SystemPrompt = [new TextBlock("You are a helpful voice assistant.")],
    Tools = [myTool],
});

IOperative operative = OperativeBuilder.Create(session).AddTool(myTool).Build();

await foreach (SessionOutputEvent e in operative.RunAsync())
{
    switch (e)
    {
        case SessionReadyEvent: Console.WriteLine("connected"); break;
        case TranscriptDeltaEvent t: Console.WriteLine($"{t.Role}: {t.Text}"); break;
        case AudioOutputDeltaEvent a: /* play a.Audio (24kHz PCM16) */ break;
        case InterruptedEvent: /* stop playback immediately */ break;
    }
}

await operative.SendAsync(new AppendAudioInputEvent { Audio = micChunk });

Fidelity notes

• Capabilities: AudioInProcess | ToolCalling | Transcripts — audio flows through this same WebSocket, and there's no native session resumption (SessionCapabilities.NativeResumption/ClientAssistedResumption are both unset).
• Tool results: the full ToolResult content (all blocks, not just text) is flattened into OpenAI's single-string function_call_output — text blocks contribute their text verbatim, anything else is JSON-serialized rather than silently dropped.
• Tool choice: RealtimeSessionOptions.ToolChoice is sent as OpenAI's own tool_choice at connect time (same shape as Chat Completions/Responses — a bare string for auto/required/none, a structured object to force one specific tool).
• Raw passthrough: OpenAI's Realtime API has substantially more event types than are mapped to canonical events here (item lifecycle events, rate-limit updates, built-in MCP tool events, text-modality deltas). Anything not explicitly mapped arrives as a RawPassthroughEvent rather than being dropped.
• Mid-session reconfiguration: OpenAI's session.update (changing instructions/voice/tools/tool-choice after connecting) has no canonical SessionInputEvent yet — unconfirmed whether Gemini or Nova Sonic support anything equivalent, so it isn't promoted. This adapter recognizes RawSendEvent for it two ways: pass a constructed SessionUpdateEvent (from Universal.OpenAI.Realtime.Client) as Payload for full control, or just set RawJson to a {"type":"session.update",...} string — the adapter deserializes that one specific shape directly (no polymorphism involved, so this only recognizes that one allowlisted type; anything else needs the typed Payload). Any other unhandled input event still throws NotSupportedException.