Bolt.Net.Media 1.0.0-dev.18

Bolt Media Streaming — Voice/Video Calls over Bolt Protocol

Overview

Bolt Media extends the Bolt binary RPC protocol with real-time audio/video call support. Unlike WebRTC, Bolt handles media transport natively — no browser dependency, no STUN/TURN infrastructure, and full server-side access to media streams for recording, transcription, and AI processing.

Current status: Phase 1-5 implemented, critical bugs fixed, 37 tests passing.

Architecture

Caller                    Bolt Hub (SFU)                    Callee
  |                           |                               |
  |-- CallSignal(Initiate) -->|-- CallSignal(Initiate) ------>|
  |<-- CallSignal(Ring) ------|                               |
  |                           |<-- CallSignal(Answer) --------|
  |<-- CallSignal(Answer) ----|                               |
  |                           |                               |
  |-- MediaConfig (audio) --->|-- MediaConfig (audio) ------->|
  |-- MediaConfig (video) --->|-- MediaConfig (video) ------->|
  |                           |                               |
  |== MediaFrame (audio) ====>|== MediaFrame (audio) ========>|
  |== MediaFrame (video) ====>|== MediaFrame (video) ========>|
  |<== MediaFrame (audio) ====|<== MediaFrame (audio) ========|
  |<== MediaFrame (video) ====|<== MediaFrame (video) ========|
  |                           |                               |
  |                      Media Tap                            |
  |                    (non-blocking copy)                    |
  |                           |                               |
  |                    IMediaProcessor                        |
  |                  (recording, transcription, AI)           |

Group Calls (SFU Mode)

  Participant A ──MediaFrame──> Bolt Hub ──MediaFrame──> Participant B
  Participant B ──MediaFrame──> Bolt Hub ──MediaFrame──> Participant A
  Participant C ──MediaFrame──> Bolt Hub ──MediaFrame──> Participant A
                                         ──MediaFrame──> Participant B

The hub forwards each participant's media to all other participants (zero-decode, raw byte forwarding).

Wire Protocol

Frame Types

MediaFrame Header (30 bytes)

[1:type=0x21] [16:streamId] [4:sequenceNumber] [4:timestamp] [1:flags] [4:payloadLen] [payload]

• sequenceNumber — monotonic per-stream, for ordering + gap detection
• timestamp — RTP-style media clock (48kHz for audio, 90kHz for video)
• flags — bit 0: keyframe, bit 1: end-of-picture, bit 2: marker, bit 3: FEC-protected, bit 4-5: priority, bit 6: drop-eligible, bit 7: compressed

CallSignal Types

Call State Machine

Initiating --> Ringing --> Active --> Ended
                       \-> Rejected   /\ (from Active or Held)
                       \-> Missed     Active <-> Held

• Initiating → Ringing: callee is online and receives the signal
• Ringing → Active: callee answers
• Ringing → Rejected/Missed: callee rejects or 30-second timeout
• Active ↔ Held: either party holds/unholds
• Active/Held → Ended: either party ends

Features

Adaptive Bitrate

The receiver sends MediaFeedback every 250ms with quality metrics:

Bitrate floor: audio 16kbps, video 100kbps. Ceiling: originally negotiated bitrate.

Forward Error Correction (FEC)

XOR-based parity. For every K source frames, one parity frame is generated.

Enabled by default on TCP (WebSocket). Dynamic: enable when loss > 3%, disable when loss < 0.5%.

Dynamic Throughput Maintenance

Multi-layer strategy to maintain target throughput under degrading networks:

Server-Side Media Hooks

public interface IMediaProcessor
{
    bool Accepts(Guid callId, MediaType mediaType);
    ValueTask ProcessFrameAsync(Guid callId, Guid streamId,
        ReadOnlyMemory<byte> frameData, uint timestamp, uint sequenceNumber);
    ValueTask OnCallStartedAsync(Guid callId);
    ValueTask OnCallEndedAsync(Guid callId);
}

// Registration
services.AddBoltServer(options =>
{
    options.MediaProcessors.Add(new RecordingProcessor());
    options.MediaProcessors.Add(new TranscriptionProcessor());
});

Codec Support

Audio

Video

Usage

.NET Client

var client = new BoltClient(serverUri, "my-service", "My App", options, logger);
await client.ConnectAsync(ct);

// Start a call
var callId = await client.StartCallAsync("other-service", video: true);

// Handle incoming calls
client.OnIncomingCall += async (info) =>
{
    await client.AnswerCallAsync(info.CallId);
};

// Send media frames
var stream = client.GetMediaStream(audioStreamId);
await stream.SendFrameAsync(opusEncodedAudio, isKeyframe: false);

// Receive media frames
await foreach (var frame in stream.ReadFramesAsync(ct))
{
    // frame.Data contains encoded audio/video
    // frame.IsKeyframe, frame.SequenceNumber, frame.Timestamp
}

// End call
await client.EndCallAsync(callId);

Browser Client (TypeScript)

import { BoltBrowserClient, AudioCodecHelper, VideoCodecHelper } from '@xframework/bolt-browser';

const client = new BoltBrowserClient('ws://bolt-hub:7000/bolt', 'browser-1', 'Browser');
await client.connect();

// Handle incoming calls
client.onIncomingCall = (callId, callerClientId) => {
    client.answerCall(callId);
};

// Start a call with media
const callId = client.startCall('other-user');
const audioStream = client.sendMediaConfig(crypto.randomUUID(), callId, true, 64);

// Encode audio via WebCodecs
const audio = new AudioCodecHelper();
await audio.initEncoder(48000, 1, 64000);
audio.onEncodedChunk = (data) => audioStream.sendFrame(data);

// Decode received audio
const decoder = new AudioCodecHelper();
await decoder.initDecoder();
audioStream.onFrame = (event) => decoder.decode(event.data, event.timestamp);

Comparison vs WebRTC

Where Bolt Media is ahead

Where WebRTC is ahead

Overall Assessment

Bolt Media: ~40-50% feature parity with WebRTC for call quality.

Bolt is ahead on server-side features (recording, transcription, AI hooks) and developer experience (.NET-native, unified protocol, simple deployment). The biggest quality gaps are P2P direct connections, media encryption, and NACK retransmission.

Implementation Phases

Remaining Work

• P2P direct connection upgrade (hub-first, then direct QUIC attempt)
• Media encryption (DTLS or application-level)
• NACK retransmission for video keyframes
• Delay-based congestion control
• Simulcast (multi-resolution encoding)
• Wire QUIC datagrams into BoltMediaStream
• Opus in-band FEC integration