Kestrel.Transport.IoUring
2.2.2
dotnet add package Kestrel.Transport.IoUring --version 2.2.2
NuGet\Install-Package Kestrel.Transport.IoUring -Version 2.2.2
<PackageReference Include="Kestrel.Transport.IoUring" Version="2.2.2" />
<PackageVersion Include="Kestrel.Transport.IoUring" Version="2.2.2" />
<PackageReference Include="Kestrel.Transport.IoUring" />
paket add Kestrel.Transport.IoUring --version 2.2.2
#r "nuget: Kestrel.Transport.IoUring, 2.2.2"
#:package Kestrel.Transport.IoUring@2.2.2
#addin nuget:?package=Kestrel.Transport.IoUring&version=2.2.2
#tool nuget:?package=Kestrel.Transport.IoUring&version=2.2.2
Kestrel.Transport.IoUring
A high-performance io_uring transport for Kestrel / ASP.NET Core that replaces the default socket transport with Linux's io_uring interface. Up to 30% faster at 128+ concurrent connections.
Features
- Drop-in replacement — one
UseIoUring()call replaces the socket transport - Multishot recv with provided buffer rings — eliminates per-recv memory pinning and SQE resubmission
- Zero-copy send (
SEND_ZC) for payloads >4KB — avoids kernel buffer copy - Registered files (
IOSQE_FIXED_FILE) — kernel skips fd lookup per SQE - Single-issuer IO loop — dedicated thread for io_uring_enter, drain tasks submit via eventfd wake
- Graceful fallback — auto-detects
ENOSYS/EPERMand falls back to socket transport - Zero-allocation hot path with pooled
IValueTaskSourcecompletions - Multi-ring support via
SO_REUSEPORT(opt-inThreadCountoption) - Targets net8.0, net9.0, net10.0
Requirements
- Linux with kernel 6.0+ (for multishot recv + buffer rings)
- Kernel 5.1+ works with reduced features (single-shot recv, no buffer rings)
- .NET 8 or later
Fallback: On unsupported systems, a warning is logged and the standard
SocketTransportFactoryis used — your app keeps working.
Installation
dotnet add package Kestrel.Transport.IoUring
Quick Start
var builder = WebApplication.CreateBuilder(args);
builder.WebHost.UseIoUring(options =>
{
options.RingSize = 256; // SQ/CQ ring depth (power of two)
options.MaxConnections = 1024; // connection limit
});
var app = builder.Build();
app.MapGet("/", () => "Hello from io_uring!");
app.Run();
IHostBuilder variant
Host.CreateDefaultBuilder(args)
.UseIoUring()
.ConfigureWebHostDefaults(web => { ... })
.Build()
.Run();
Architecture
┌──────────────────────────────────────────────────────────────────┐
│ ASP.NET Core / Kestrel │
│ IConnectionListenerFactory.BindAsync() │
│ │ │
│ IoUringTransportFactory ─── creates ──► Ring (io_uring fd) │
│ │ │ │
│ IoUringConnectionListener ◄─── IORING_OP_ACCEPT completions ──┤
│ │ │ │
│ IoUringConnection (ConnectionContext) │ │
│ ├── Input Pipe ◄──── IORING_OP_RECV ─────────┤ │
│ └── Output Pipe ─────► IORING_OP_SEND ────────┘ │
└──────────────────────────────────────────────────────────────────┘
Native layer
| File | Description |
|---|---|
Native/IoUringNative.cs |
io_uring_setup / io_uring_enter / io_uring_register via syscall() |
Native/IoUringSqe.cs |
io_uring_sqe struct (64 bytes, [StructLayout(Sequential, Size=64)]) |
Native/IoUringCqe.cs |
io_uring_cqe struct (16 bytes) |
Native/IoUringParams.cs |
io_uring_params with SQ/CQ offset sub-structs |
Native/IoUringConstants.cs |
Opcode constants, mmap offsets, flag bits |
Native/Libc.cs |
mmap / munmap / close P/Invoke |
Ring layer
| File | Description |
|---|---|
SubmissionQueue.cs |
Unsafe pointer arithmetic over mmap'd SQ ring; Volatile.Read/Write for SMP ordering |
CompletionQueue.cs |
Unsafe pointer arithmetic over mmap'd CQ ring |
Ring.cs |
Wraps SQ+CQ; TryGetSqe, Submit, SubmitAndWait, TryPeekCompletion; static IsSupported |
Transport layer
| File | Description |
|---|---|
Transport/IoUringTransportFactory.cs |
Implements IConnectionListenerFactory; creates a Ring per bound endpoint |
Transport/IoUringConnectionListener.cs |
Binds socket, submits ACCEPT SQEs, single IO-loop task, bounded Channel<ConnectionContext> |
Transport/IoUringConnection.cs |
ConnectionContext; Pipe-based IDuplexPipe; submits RECV/SEND SQEs |
Transport/IoUringTransportOptions.cs |
RingSize, ThreadCount, MaxConnections |
WebHostBuilderIoUringExtensions.cs |
UseIoUring() for IWebHostBuilder and IHostBuilder |
Performance
On a 2-core VM (AMD EPYC 9V74, Linux 6.17, .NET 10):
| Scenario | Socket | io_uring | Improvement |
|---|---|---|---|
| 128 persistent connections | 17,897 req/s | 23,488 req/s | +31% |
| 256 persistent connections | 18,800 req/s | 23,785 req/s | +27% |
| 20K requests @ 128 concurrency | 30,868 req/s | 35,837 req/s | +16% |
io_uring's advantage grows with connection count. On multi-core servers, the improvement is larger.
See BENCHMARKS.md for detailed results and methodology.
Project Structure
├── src/Kestrel.Transport.IoUring/ ← NuGet library (net8.0/net9.0/net10.0)
├── samples/SampleApp/ ← Minimal ASP.NET Core sample
└── benchmarks/ ← Stress benchmarks
io_uring Features Used
| Feature | Kernel | Purpose |
|---|---|---|
IORING_OP_ACCEPT |
5.1+ | Accept new TCP connections |
IORING_OP_RECV (multishot) |
6.0+ | Receive data with buffer ring selection |
IORING_OP_SEND |
5.1+ | Send data (small payloads) |
IORING_OP_SEND_ZC |
6.0+ | Zero-copy send (payloads >4KB) |
IORING_OP_READ |
5.1+ | eventfd wake for IO loop |
IORING_REGISTER_FILES |
5.1+ | Registered file descriptors |
IORING_REGISTER_PBUF_RING |
5.19+ | Provided buffer ring for recv |
IORING_FEAT_SINGLE_MMAP |
5.4+ | Shared SQ/CQ mmap region |
User Data Encoding
Each SQE carries a user_data field used to route the CQE back to the correct connection and operation:
bits [63..8] = connection_id (int, shifted left 8)
bits [7..0] = op_type (0=Accept, 1=Recv, 2=Send, 3=Close)
License
MIT
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- No dependencies.
-
net8.0
- No dependencies.
-
net9.0
- No dependencies.
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
v2.2.1 — Slot free-list: fixes high-c correctness + unlocks stronger wins
• Fix: per-listener connection-slot allocator now uses a pre-populated free-list
(Stack<int>) instead of the pre-2.2.1 `connId % MaxConnections` modulo. The
modulo policy silently aliased connIds to slot indices and overwrote live
connections' slots once connId count exceeded MaxConnections, orphaning their
recv/send-loop/fd and producing mass errors at connection counts > MaxConnections.
Default MaxConnections is now 16384 (was 1024). Slot release is deferred until
all in-flight CQEs for the closing connection have drained, so the generation
check correctly rejects stale CQEs instead of misrouting them to a new
connection that happened to claim the same slot.
• Fix: HandleAccept now soft-rejects (close fd, log warning) when the free-list
is empty, instead of the previous silent-overwrite. No more orphaning under
bursts above MaxConnections.
Benchmarks (dotnet/Benchmarks aspnet-gold-lin, 56-core Intel Xeon Gold 6330,
Ubuntu-24.04 kernel 6.8, IOURING_THREAD_COUNT=32, 1-trial-per-cell matrix):
json c=128: 1.07M RPS vs sockets 1.04M (+0.7% RPS, -33% CPU)
json c=512: 979K RPS vs sockets 937K (+4.5% RPS, -38% CPU)
json c=1024: 1.08M RPS vs sockets 1.04M (+3.9% RPS, -37% CPU)
json c=2048: 1.04M RPS vs sockets 958K (+8.8% RPS, -46% CPU) *pre-2.2.1: 401K errors*
plaintext-dual c=1024: 15.45M RPS (+0.9% RPS, -40% CPU)
plaintext-dual c=2048: 15.06M RPS (-1.3% RPS, -52% CPU) *pre-2.2.1: 719K errors*
tls-steady-dual c=128: 798K RPS (+13.5% RPS, -41% CPU)
tls-steady-dual c=1024: 1.21M RPS (+7% RPS, -35% CPU)
tls-steady-dual c=2048: 1.22M RPS (+9.5% RPS, -41% CPU)
0 errors across all 24 cells (12 scenarios/connection pairs × 2 variants).
Known limitation (unchanged from 2.2.0): bombardier TLS handshake-churn with
`connectionclose=true` still hits an IO-loop livelock after the first close.
This is orthogonal to the slot-collision bug — reproduces at c=1 and with
IOURING_MAX_CONNECTIONS=65536. Deferred; keep-alive workloads (the common
production shape) are unaffected.
v2.2.0 — Stability + correctness + smaller pinned heap; consistently beats sockets
• S0 Correctness fixes (3 latent bugs that caused ~1429 socket errors / 60s @ c=2048
with v2.1.0 and full death-spiral at higher loads):
- Accept-channel drop: when bounded accept-channel was full under burst, the new
connection was created but TryWrite return value was ignored, leaking the conn.
Now: on TryWrite=false, abort + close cleanly.
- Multishot recv rearm SQ-full silent loss: SubmitMultishotRecv now returns bool;
callers add to the recv-retry set on failure. No more silent stop-receiving.
- CQ-overflow recovery: RecoverFromCqOverflow now routes via UsingMultishotRecv
flag; previously always used single-shot recv even for buffer-ring connections.
• S1 Skip per-connection pinned recv buffer when buffer ring is active (it's never
used in that path). Saves ~4 KB POH per connection.
• P1 Default ThreadCount tuning: TC=64 was wildly oversubscribed on 56-core boxes.
TC=32 (or matching ProcessorCount/2) gives same/better RPS at half the CPU.
• P2 Buffer-ring right-size: ReceiveBufferSize default 4096 -> 2048. Halves the
per-buffer pinned heap; keep-alive HTTP/1.1 small payloads need ~2 KB.
• S2 Diagnostics: new RecvDiagnostics counters (ENOBUFS, EPIPE, ECONNRESET, accept
channel drops, recv-retry queue depth/water-mark, async-flush pending). Periodic
log via LogPoolStatsInterval.
• Result: stable to c=4096+ (zero socket errors, zero bad responses, no death
spiral). v2.1.0 is broken at c>=256 — v2.2.0 fixes that AND beats sockets.
Benchmarks (json, 56-core Intel Xeon Gold 6330, Azure D64as_v5, kernel 6.8, 3-trial
median, single-load except c=2048 which is dual-load 2x c=1024):
c=256 patched 781,942 RPS @ 28% CPU vs sockets 753,172 @ 40% CPU
+3.8% RPS, -30% CPU, +48% RPS/CPU%, P99 0.64 vs 0.70 ms, 0 errors
c=1024 patched 1,068,612 RPS @ 34% CPU vs sockets 1,046,915 @ 51% CPU
+2.1% RPS, -33% CPU, +53% RPS/CPU%, P99 5.50 vs 6.41 ms, 0 errors
c=2048 patched 2,132,290 RPS @ 34% CPU vs sockets 2,037,381 @ 48% CPU
+4.7% RPS, -29% CPU, +48% RPS/CPU%, P99 5.74 vs 6.55 ms, 0 errors
c=4096 patched 1,718,626 RPS @ 33% CPU, P99 3.59 ms, 0 errors (stress validation)
Pinned heap dropped from 75 MB (v2.1.0 at TC=64) -> 16 MB (4.7x reduction).
v2.0.0 — Architecture overhaul: beats sockets by 15%
• IoUringPipeScheduler: custom PipeScheduler runs all pipe continuations on
the IO loop thread. Eliminates ThreadPool hops for both recv and send paths.
• Inline HTTP processing: Kestrel HTTP pipeline runs on the IO thread
(like UnsafePreferInlineScheduling). Zero cross-thread signaling.
• Non-blocking send: IValueTaskSource-based send completion. Send loop yields
after SQE submit, resumes inline on CQE. Zero-alloc send path.
• Connection slot array: replaces ConcurrentDictionary with O(1) direct index.
• Pre-pinned recv buffer: eliminates per-recv Pin()/Dispose().
• Fix SetSqeFd infinite recursion bug (stack overflow when fd registration fails).
• Coalesced PipeScheduler wakeups: flag-based eventfd dedup.
Benchmark: json scenario, 56-core Intel Xeon Gold 6330, kernel 6.8.0
256c: 2,043K RPS (+9% vs sockets)
512c: 2,499K RPS (+13% vs sockets)
1024c: 2,829K RPS (+15% vs sockets)
CPU usage: 50% (sockets: 90%) — 1.9x more efficient per CPU cycle
Lock contention: 50/s (sockets: 1,994/s)
GC time: 0% | Allocation rate: 3.8M B/s (below sockets)
v1.2.0 — Re-enable buffer ring + SQPOLL mode
• Re-enable provided buffer ring (multishot recv) — stalling bug fixed
on kernel 6.6+. Eliminates per-recv Pin()/Dispose() overhead.
Controlled via EnableBufferRing option (default: true).
• Add SQPOLL mode support (EnableSqPoll option). Kernel polls the SQ ring,
eliminating io_uring_enter syscalls. Requires CAP_SYS_NICE or root.
• Ring constructor now accepts setup flags for SQPOLL and future kernel features.
• BufferRingSize option for tuning buffer ring depth (default: 256).
v1.1.0 — Performance: lock-free send path
• Send operations are now enqueued to the IO loop via ConcurrentQueue + eventfd
instead of acquiring SubmitLock per send. Eliminates 90K+/s lock contentions
on multi-core machines.
• IO loop is now the sole SQ producer — all submissions are batched into a
single io_uring_enter per loop iteration.
• Reduces per-send syscalls from 1 each to 0 (batched with recv/accept work).
v1.0.1 — Bug fix
• Fix IOSQE_BUFFER_SELECT constant: was 1<<3 (0x08), now 1<<5 (0x20)
Matches kernel definition. Fixes EFAULT when using buffer rings with recv.
• Add test project with 10 tests (unit + integration)
• Add InternalsVisibleTo for test project
v1.0.0 — Initial release
• Drop-in io_uring transport via UseIoUring()
• Multishot recv with buffer rings
• Zero-copy send (SEND_ZC)
• Registered file descriptors
• Targets net8.0, net9.0, net10.0