ClickHouse.Driver 1.4.0-rc2

• net10.0

  • Microsoft.Extensions.Http (>= 10.0.8)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.8)
  • Microsoft.IO.RecyclableMemoryStream (>= 3.0.1)
  • NodaTime (>= 3.2.3)

• net6.0

  • Microsoft.Extensions.Http (>= 10.0.8)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.8)
  • Microsoft.IO.RecyclableMemoryStream (>= 3.0.1)
  • NodaTime (>= 3.2.3)
  • System.Text.Json (>= 10.0.5)

• net8.0

  • Microsoft.Extensions.Http (>= 10.0.8)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.8)
  • Microsoft.IO.RecyclableMemoryStream (>= 3.0.1)
  • NodaTime (>= 3.2.3)
  • System.Text.Json (>= 10.0.5)

• net9.0

  • Microsoft.Extensions.Http (>= 10.0.8)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.8)
  • Microsoft.IO.RecyclableMemoryStream (>= 3.0.1)
  • NodaTime (>= 3.2.3)
  • System.Text.Json (>= 10.0.5)

v1.4.0
---

**Breaking Changes:**
* **Query responses are now compressed with zstd instead of gzip by default.** The driver advertises `zstd, lz4, gzip, deflate`, and ClickHouse's fixed preference order resolves that to zstd. Bodies are decoded transparently as before, so no code change is needed.
* **Binary inserts are now compressed with zstd instead of gzip by default.** `InsertOptions.Compressor` defaults to `ZstdCompressor.Default` (level 3) rather than `GZipCompressor.Default` (gzip, fastest). You can control this via `InsertOptions`.

* **If you set `AcceptEncoding` and then read a raw result, you now get compressed bytes.** `ExecuteRawResultAsync`, `PostStreamAsync` and `InsertRawStreamAsync` hand back the response body exactly as the server sent it. Call `ClickHouseRawResult.ReadDecompressedStreamAsync()` to have the driver decompress it, or check `ContentEncoding` and decompress it yourself. Raw exports with no `AcceptEncoding` set are unaffected: plaintext before, plaintext now.
* **Reading a column value from `ClickHouseDataReader` with no current row now throws `InvalidOperationException`** — that is, before the first `Read()` or after `Read()` has returned `false`.

**New Features:**
* Added `InsertOptions.QueryPlacement`. Set it to `InsertQueryPlacement.Url` to send a binary insert's `INSERT` statement as the `query` URL parameter, where proxies and access logs can read it, instead of in the request body; body placement remains the default. If the .NET runtime rejects an oversized URL, the error directs callers back to body placement. (#580)
* Pluggable binary-insert compression via `InsertOptions.Compressor` (`IClickHouseCompressor`). The presence of a compressor is the on/off switch: set it to `null` to send the payload uncompressed (useful over fast/local links where the compression CPU outweighs the bandwidth savings). GZip, Brotli, LZ4, and ZStd compressors are available.
 - **Tuning guidance:** the best codec depends on where the server is. Over a fast/local link, compression is pure overhead, so `Compressor = null` is worth trying. Over a remote/cloud connection the payload reduction dominates, depending on bandwidth.
* Built-in LZ4 codec for binary inserts (`Lz4Compressor`; HTTP `Content-Encoding: lz4` plus the native-protocol block path). LZ4 now ships **in the core driver with no third-party runtime dependency**.
* Built-in ZSTD codec (`ZstdCompressor`; HTTP `Content-Encoding: zstd` plus the native-protocol block path).
* You can now choose how ClickHouse compresses query responses, via `AcceptEncoding` — client-wide (`ClickHouseClientSettings.AcceptEncoding`), in the connection string (`AcceptEncoding=lz4`), or for one query (`QueryOptions.AcceptEncoding`). `lz4`, `gzip`, `deflate` and `br` are all decompressed for you; `identity` turns response compression off. Previously only `gzip` and `deflate` worked: asking for `lz4` or `br` returned unreadable data, and `identity` did not actually switch compression off. To decompress a raw export, use the new `ClickHouseRawResult.ReadDecompressedStreamAsync()`.
* zstd responses are now decoded transparently, so `AcceptEncoding = "zstd"` works with every read API instead of failing as an unsupported codec.
* Added `MapReadMode` (connection string `MapReadMode=KeyValuePairs`, or `ClickHouseClientSettings.MapReadMode`), which reads `Map(K, V)` columns as `List<KeyValuePair<K, V>>` instead of `Dictionary<K, V>`. A ClickHouse map may repeat a key and a dictionary cannot, so the default representation silently drops pairs. Also accepted on the write path.

**Improvements:**
* A large number of performance improvements have been made in this version, significantly cutting heap allocations, reducing GC pressure, and improving speed. You will see the best results by using the POCO read and write methods, as they avoid boxing values.
* **A custom `HttpClient` no longer needs `AutomaticDecompression`.** The driver decompresses responses itself, so supplying your own `HttpClient` (or `IHttpClientFactory`) no longer risks unreadable data. If you do set `AutomaticDecompression`, prefer leaving it at its default of `None`.
* **Clearer failures around compression.** A response compressed with a codec the driver cannot read (e.g. `snappy`) now raises an error naming that codec and how to fix it, instead of failing as a confusing type-parse error. Server error messages are now readable when the response is compressed, rather than surfacing as binary.
* Lower GC pressure on binary inserts: the built-in compressors (`GZipCompressor`, `BrotliCompressor`, `Lz4Compressor`) no longer allocate a fresh 256 KiB `BufferedStream` buffer per batch on the large object heap. The buffering now rents its backing buffer from `ArrayPool<byte>`.
* Binary inserts (`InsertBinaryAsync`) now stream the serialized batch directly into the HTTP request body instead of buffering the whole payload. Requests are now sent with chunked transfer-encoding (no `Content-Length`).
* Removed a 4KiB/batch allocation in the binary-insert serializers (for writing the SQL query). Pooled buffers are now used instead.
* Faster multidimensional array binary inserts: rectangular multidimensional CLR arrays (`int[,]`, `double[,,]`, …) of fixed-width primitive leaves (`Int8/16/32/64`, `UInt8/16/32/64`, `Float32/64`, `Bool`) are now serialized by blitting each contiguous inner row in a single write.
* Reduced allocations on the binary write path. Writing an `Int128`/`UInt128`/`Int256`/`UInt256` value no longer allocates two temporary arrays per value, and a rectangular multidimensional array whose leaf is a fixed-width primitive behind a wire-transparent wrapper (`LowCardinality(Int32)`, `SimpleAggregateFunction(any, Int32)`, …) now takes the same blit fast path as a bare leaf instead of boxing every element. `Nullable(...)` leaves keep the per-element path. Output bytes are unchanged.
* Reduced allocations when writing `Decimal`/`Decimal128`/`Decimal256` values (binary inserts and parameters): the write path no longer allocates a `BigInteger.ToByteArray()` array plus a separate destination buffer per value.
* Mostly eliminated allocations in POCO binary inserts: value-type properties are now written through box-free delegate instead of boxing each value into `object` and unboxing it inside the type's `Write`. Note that if you use `RowBinaryWithDefaults`, the path is unchanged and still incurs the boxing cost. The change only applies to `RowBinary` (the default).
* Faster `Variant` write-type resolution: matching a value to its variant subtype now uses an O(1) hash lookup by runtime type instead of an O(n) linear scan for variants with 3 or more underlying types.
* Lower GC pressure when reading query responses: the reader no longer allocates a fresh `BufferedStream` buffer per query. The response read buffering now rents its backing buffer from `ArrayPool<byte>` and returns it when the reader is disposed. **The default `ReadBufferSize` has been raised from 8 KiB to 64 KiB** now that pooling removes the per-query allocation that motivated the small default (see v1.3.0), reducing read refills on large responses.
* Faster scalar reads from query responses: the response stream wrappers now implement `Stream.Read(Span<byte>)` directly instead of falling through to the base implementation, which rents and copies through a pooled array on every call, and mid-stream exception detection now records from below the read buffer so it observes one read per buffer refill rather than one per value decoded.
* Reduced per-query request-URI allocations: `ClickHouseUriBuilder.ToString()` now composes the URI into a thread-reused `StringBuilder` instead of allocating a `UriBuilder`, two dictionaries, and a LINQ-projected string per parameter. Cuts URI-assembly allocations ~65% on the common parameter-free path. The produced URI string and behavior are unchanged.
* Reduced per-row allocations when reading fixed-size binary columns. `Int128`/`Int256`/`UInt128`/`UInt256`, `Decimal128`/`Decimal256`, `IPv4`, and `IPv6` no longer allocate a temporary `byte[]` per value during deserialization.
* Reading an `Int128` or `UInt128` column into a native `Int128`/`UInt128` property no longer allocates a 16-byte array per value (40 bytes per value on 64-bit).
* Reduced the per-value allocations on the read and write paths for `UUID`, `Array(...)` and `Decimal` columns.
* Lower GC pressure when reading `Dynamic` columns: every value in a `Dynamic` column carries its own binary type header, so the driver decoded (and allocated) a fresh type descriptor per row. Stateless, parameterless types now decode to a shared immutable singleton instead, eliminating the per-row type allocation.
* Faster reading of `Tuple(...)` columns with up to 7 elements: `TupleType.Read` now constructs the `System.Tuple<...>` directly through a cached compiled factory instead of allocating two intermediate `object[]` buffers and invoking the constructor reflectively via `Activator.CreateInstance`.
* Faster reading of `Array` columns with common leaf element types (integers, floats, `Bool`, `Decimal`, date/time types, `UUID`, `String`, `Enum`, `FixedString`, and their `Nullable` forms): `ArrayType.Read` now fills a strongly-typed `T[]` through the array indexer instead of allocating via `Array.CreateInstance` and storing each element with reflective `Array.SetValue`.
* Reading `Map` columns now pre-sizes the result `Dictionary` to the known entry count.
* Reduced allocations in POCO reads (`client.QueryAsync<T>(...)`): scalar, `String` and `FixedString` columns are now materialized straight into the target property instead of through a boxed `object[]` row buffer, removing one box and one unbox per value-type property per row.
* `SimpleAggregateFunction(f, T)` columns now take the box-free fast path in `QueryAsync<T>`, as `LowCardinality(T)` already did. Previously they fell back to the slower boxed read.
* `QueryAsync<T>` can now bind one column to more than one CLR type, chosen by the property: `DateTime`/`DateTime64`/`Date` into `DateTime`, `DateTimeOffset` or `DateOnly`; `String`/`FixedString` into `string` or `byte[]`; `Decimal` into `decimal` or `ClickHouseDecimal`; `Enum8`/`Enum16` into `string` (the label) or `int` (the stored numeric value); and, on .NET 8+, `Int128`/`UInt128` into the native `System.Int128`/`System.UInt128` as well as `BigInteger`.
* `client.QueryAsync<T>(...)` can now read a `Map(K, V)` column into a `List<KeyValuePair<K, V>>` or `KeyValuePair<K, V>[]` POCO property, in addition to `Dictionary<K, V>`.
* With an `IReadValueConverter` configured, `client.QueryAsync<T>(...)` applies it per column on the overload that matches how the column was read: `ConvertValue<T>`, with `T` the property type, for a column read without boxing, and the boxed `ConvertValue` for a composite one.
* Reduced per-row allocations for `ClickHouseDataReader` typed accessors (`GetInt64`, `GetDouble`, `GetDateTime`, `GetGuid`, …), `GetFieldValue<T>`, `IsDBNull` and ORMs such as linq2db by decoding scalar columns into reusable typed slots instead of eagerly boxing every value-type cell. Allocations drop by roughly two thirds on a typical multi-column typed read and to zero per row for all-numeric typed reads.
* With an `IReadValueConverter` configured, `ClickHouseDataReader`'s typed accessors (`GetInt64`, `GetDouble`, `GetDecimal`, `GetString`, …) keep their reduced allocations: each reads its typed slot and converts through `ConvertValue<T>`, the overload `GetFieldValue<T>` uses. These accessors previously routed through `GetValue` and so through the boxed `ConvertValue`.
* Added `ClickHouseDataReader.TryGetEnumOrdinal(int ordinal, out int value)` for reading the underlying integer ordinal of an `Enum8`/`Enum16` (or `Nullable(Enum...)`) column.
* You can now pass `UseFormDataParameters` through the connection string. Added `UseFormDataParameters` property to `ClickHouseConnectionStringBuilder`.
* Reading `Map` columns now builds the result `Dictionary` through a cached compiled factory instead of a reflection invoke, removing the per-row constructor resolution that the pre-sizing change added.

**Deprecations:**
* `ClickHouseClient.MemoryStreamManager` is now `[Obsolete]`. Since binary inserts stream directly into the request body (see above), this property is no longer used and has no effect; it will be removed in a future version.

**Bug Fixes:**
* **Breaking Change**: Fixed a `JSON` column with overlapping paths silently losing data. ClickHouse accepts a column which declares a path both as a value and as the parent of another path — for example `JSON(a Int64, a.b Int64)` — and sends both paths in every row, so the row renders as a document with a duplicate key (`{"a":0,"a":{"b":7}}`). A `JsonObject` cannot hold two values for one key, so the driver silently kept whichever path the server sent last: reading `{"a":{"b":7}}` back out of that column returned `{"a":0}` and the `7` was gone. The same loss hit a `Map` path overlapped by a deeper path (`JSON(a Map(String, Int64))` holding `{"a":{"b":1},"a.b":7}` returned `{"a":{"b":7}}`, dropping the map's own entry), except that there the two values were merged into one object rather than one replacing the other. A row where both sides of such an overlap hold a value now throws a `SerializationException` naming the two paths. With non-`Nullable` paths the driver cannot tell a stored `0` from an absent path, so no rule can recover the real value — the choice is between a wrong answer and a clear failure. A side which holds nothing is not a collision: a null, an empty object, and an all-null subtree all give way to the side which has the data, whichever of the two paths the server sends first, so `Nullable` overlaps continue to read as before. Use `JsonReadMode.String` to read such a column as the server's JSON text, duplicate key included. To restore the previous behavior and keep reading the column as a `JsonObject`, set the new `AllowDuplicateJsonKeys` setting (connection string key `AllowDuplicateJsonKeys=true`, or `ClickHouseClientSettings.AllowDuplicateJsonKeys`); the driver then keeps whichever value the row carries last, with the data loss described above. A path which holds a value and whose parent holds a scalar or an array still throws — that shape raised `InvalidCastException` before and is now a `SerializationException`. (#583)
* Fixed uncompressed binary inserts (`InsertOptions.Compressor = null`) inflating the HTTP request body. The request body is sent chunked, and the row serializer wrote straight to it, so every field write became its own HTTP chunk.
* Fixed binary inserts flushing the compression stream once per value written by copying a stream — binary-mode `JSON` POCOs, and `String`/`FixedString` values given as a `Stream`.
* Fixed `InsertOptions.WithColumnTypes()` and `InsertOptions.WithQueryId()` silently dropping some caller-set options (such as `AcceptEncoding`) when copying.
* Fixed POCO binary inserts writing `JSON` columns against the wrong typed-path hints when the same POCO type is inserted into two tables whose `JSON` columns declare different hints. The second insert reused the first table's cached write delegates, so values were written silently mis-typed.
* Fixed `ClickHouseServerException` carrying a blank `Message` and an `ErrorCode` of `-1` when the server — or an upstream component such as a load balancer or the ClickHouse Cloud edge — returned a non-2xx HTTP response with an empty (or whitespace-only) body. The exception now reports the HTTP status code and reason phrase, and uses the `X-ClickHouse-Exception-Code` response header as the error code when the server sets it. Non-empty error bodies are unaffected.
* Fixed `ClickHouseClient` leaking the query's `HttpResponseMessage` on the paths that fully consume it (`ExecuteNonQueryAsync`, `InsertBinaryAsync`) and on any request that fails. Paths that hand the response to the caller (`ExecuteReaderAsync`/`ExecuteRawResultAsync`/`InsertRawStreamAsync`/`PostStreamAsync`) are unchanged.
* Fixed `ClickHouseClient.InsertRawStreamAsync` disposing the supplied `Stream` twice when the request failed: the request message already disposes the content it carries, so the extra `Dispose()` in the failure path was redundant. The stream-taking write paths now document that the supplied stream is disposed once the request completes.
* Fixed `PingAsync` ignoring the `Path` connection setting: the ping request now targets `<Path>/ping`, so clients behind a reverse-proxy prefix no longer report a healthy server as unreachable.
* Fixed mid-stream server exceptions never surfacing on the streaming read path (`ExecuteReader`/`ExecuteReaderAsync`). A query that fails after the HTTP response is committed (for example a `throwIf` partway through a large result) now raises a `ClickHouseServerException` with the real server error, instead of a bare `HttpIOException` or `EndOfStreamException`.
* Fixed `ClickHouseCommand` returning wrong results (or a syntax error) for `CommandBehavior.SchemaOnly` and `CommandBehavior.SingleRow` when `CommandText` ended with a single-line comment (`--` / `#`) or a statement-terminating `;`.
* Fixed `ClickHouseDataReader.GetSchemaTable()` leaving `NumericScale` unset (`DBNull`) for `DateTime64(N)` and `Time64(N)` columns (including their `Nullable(...)` variants). The schema table now reports the fractional-seconds precision `N` in `NumericScale`, matching how `Decimal` columns are already reported.
* Fixed `DbConnection.GetSchema("Columns", ...)` not disposing the command it creates internally, which delayed the release of that command's cancellation-token source until garbage collection.
* Fixed `ClickHouseConnection.GetSchema("Columns", ...)` building invalid SQL when the table restriction is supplied without the database restriction (for example `[null, "functions"]`). The `WHERE` clause is now composed from the restrictions that are actually set, so filtering by table alone no longer fails with a server syntax error.
* Fixed `ClickHouseConnection.GetSchema("Columns", ...)` silently ignoring restriction values beyond the supported `database` and `table` positions.
* Fixed JSON typed paths whose names start with `max_dynamic_paths` or `max_dynamic_types` being mistaken for JSON settings and decoded as dynamic values.
* Fixed `JSON` columns being unreadable when a typed path name requires backtick quoting — for example ``JSON(`a b` Int64)`` or ``JSON(`a,b` Int64)``. Such a path made the whole query fail with `SerializationException: Unsupported path in JSON hint`, because the type parser split each hint on every space and did not treat backticks as quotes. Quoted path names (including ones containing spaces, commas, parentheses and escaped characters) are now parsed and unescaped correctly.
* Fixed reading a `JSON` column where a typed path holds `NULL`. The path was dropped from the returned `JsonObject` entirely, so `{"x": null}` came back as `{}` and callers could not tell "path not present in this row" from "path present but null"; for a nested typed path such as `JSON(a.b Nullable(Int64))` the whole parent subtree disappeared. Typed paths are now materialized with an explicit JSON null, matching the server's own JSON rendering. Dynamic (unhinted) paths are unchanged and stay absent, as the server also omits them.
* Fixed string values inside a `JSON` column being returned as base64 when `ReadStringsAsByteArrays = true`, and `Map(String, ...)` keys throwing `InvalidCastException`. String leaves inside a `JSON` column are now always decoded as UTF-8 text regardless of the setting — remove any base64 workaround; the setting is unchanged for ordinary `String`/`FixedString` columns.
* Fixed named `Tuple` and `Nested` columns being unreadable when an element name requires backtick quoting and contains a space — for example ``Tuple(`p q` Int64, r String)`` or ``Nested(`a b` Decimal(10, 2), c String)``. Reading such a column failed with `ArgumentException: Unknown type`, because the type parser split the element declaration on its first space and so cut the quoted name in half. The element name is now skipped as a whole before the name/type separator is located.
* Fixed the type parser building a self-referential node for an empty parameter list, such as the `Tuple()` column type that ClickHouse accepts and reports back in `system.columns`.
* Fixed enum type names rendering as invalid ClickHouse syntax. Enum labels are now quoted and escaped, and the declaration includes its closing parenthesis.
* Fixed HTTP query parameters mangling a `byte[]`/`ReadOnlyMemory<byte>` bound to `String`/`FixedString` (a `byte[]` was sent as the literal text `System.Byte[]`) and rejecting a `TimeOnly` bound to `Time`/`Time64`. Byte payloads are now escaped byte-for-byte, so data that is not valid UTF-8 round-trips losslessly; `TimeOnly` binds on both the HTTP and binary write paths, and infers as `Time64(7)` when no type hint is given.
* Fixed `{name:Type}` parameter type hints being mis-detected in queries containing `//` comments, nested block comments, backtick/double-quoted identifiers, backslash escapes or `$tag$` heredocs. A bare `#` no longer starts a comment (only `# ` and `#!` do).
* Fixed `{name:Type}` parameter type hints being dropped, or a hint being invented for a parameter that does not exist, when the query contains another `{` that is not a type hint — for example a `SETTINGS` map value such as `additional_table_filters = {'t': 'a > 0'}`. A dropped hint fell back to CLR-type inference, losing precision.
* Fixed ADO-style `@name` parameter placeholders being rewritten inside string literals, quoted identifiers, heredocs and comments, which corrupted values such as `'user@id'` into `'user{id:Int32}'`. Placeholders are now only replaced in code positions.
* Fixed a `$` inside an unquoted identifier being mistaken for the start of a `$tag$` heredoc. Everything up to the next occurrence of the same `$...$` text was skipped as heredoc body, silently dropping any `{name:Type}` type hint (and any ADO-style `@name` placeholder) in between — for example in `WITH 1 AS b$c$ SELECT {d:Date} AS v, b$c$ AS x`. A heredoc is now only recognized where a token starts, matching the server lexer.
* Fixed ADO-style `@name` placeholders not working when the parameter name contains a `$`, which ClickHouse accepts in a query parameter name: `@id$x` could not be bound at all (the name was interpolated into a regex, where `$` is an end-of-input anchor), and a shorter name won over a longer one, so with only `id` defined `SELECT @id$x` was silently rewritten into a different, still valid query that aliased the value as `$x` instead of being left for the server to reject. A `$` is now part of the placeholder name, matching the server lexer.