IndxSearchLib 5.0.2

• net10.0

  • CommunityToolkit.HighPerformance (>= 8.4.0)
  • Microsoft.Data.Sqlite (>= 10.0.3)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.3)
  • SQLitePCLRaw.bundle_e_sqlite3 (>= 3.0.3)

Vector search rebuilt, and the last places where a wrong argument was answered with an
     exception or with nothing. Scores and order of vector and hybrid results change, the `HNSW`
     package is no longer a dependency, and `TrySetFieldConfiguration` returns a refusal where it
     threw: see the entries marked Changed before upgrading.
     
     - Changed: **vector search finds what it should.** The graph behind a vector field is now the
       library's own, a port of FAISS's HNSW with hnswlib's deletion, and the `HNSW` package (and the
       `MessagePack` pin it needed) is no longer a dependency. Measured on real embeddings against
       the exact nearest: the ten returned held 94.5 % of the true ten where the old graph held about
       half, by a default of the package that linked each vector only to its nearest. The same
       figures as FAISS on three datasets, a build two and a half to five times faster than before,
       and 140 bytes of graph a vector. Scores and order of vector and hybrid results change.
     - Changed: **a vector is brought to unit length**, when stored and when searched for, so
       `IEmbeddingIndex.Search` returns the cosine whatever length the vectors have. They were taken
       to be unit length already and the dot product returned as it was.
     - Added: **`ConfigurationParameters.EmbeddingSetup`**, with `GraphVectors`: how the search graph
       holds its vectors, `VectorStorage.Int16` (the default) or `Float32`. 16-bit integers halve the
       graph's vectors in memory and make a search and a build about a third faster; recall was
       unchanged to the third decimal and a score differs in the fifth. It decides nothing a client is
       handed. A new constructor overload and a new `With` argument; the existing constructors take
       the default, and a configuration file written before it is read as the default.
     - Changed: **a document with a vector takes a third of the memory, and its vector's numbers
       may come back spelled differently.** `EmbeddingSetup.DocumentVectors` says how a document's
       own copy of a vector is held: `DocumentVectorStorage.Float32` (the default), `Int16` or
       `Text`. As `Float32` the vector is taken out of the document's JSON when it is stored and
       written back when the document is read: the same numbers, in the shortest text that reads
       back as the same float, so `0.10` comes back `0.1` and a number with more digits than a float
       holds comes back with as many as it holds. A client that compares documents as text sets
       `Text`, which gives every document back character for character. `Int16` halves the vector
       once more and changes its numbers in the fifth or sixth decimal. Measured on 15 000 products
       with embeddings of 1 536 numbers: 47 KB a document as `Text`, 16 KB as `Float32`, 13 KB as
       `Int16` (before this release, with one stored position per number, about 110 KB). Reading a
       document back costs some 0.2 ms as numbers, for writing the vector out, against a microsecond
       as text. What is written to a database is the record as it arrived, whichever is set.
       A fourth value, `NotKept`, sends the vector to the search graph and keeps no copy in the
       document: a document read back has no vector field. With it, a record sent to
       `UpdateJsonRecords` without the vector field leaves the vector as it is, and `null` removes
       it. `CreateInMemoryClone` takes the vectors from the graph, and with a `Persistence` a record
       written back without its vector is stored with the graph's: the graph's copy, which on 16-bit
       integers is the original to about four digits, at the original's length.
     - Fixed: **`Status` no longer reports `Ready` before `Index()` has finished.** While the engine
       was indexing, `Status.SystemState` passed on the state of the text index underneath, which is
       ready before `Index()` is: the vector graphs that need it are rebuilt after, and the search
       buffers swapped. A caller watching the state saw `Ready` through that, with searches still
       answering `DidTimeOut`, for minutes on a large vector field. A caller waiting on the
       `ProcessMonitor` never saw it. `Ready` now means `Index()` is done.
     - Fixed: **a role or weight set straight on a `Field` is saved with its new value.**
       `Searchable`, `Sortable` and `Weight` told the configuration of the change before they made
       it, so an engine with a `Persistence` saved the value from before; `Facetable`, `Filterable`,
       `WordIndexing` and `Embeddable` told nothing, so a role set that way was gone after a
       restart. Every setter now tells after the change, and only when the value changed.
       `TrySetFieldConfiguration` saves once for the whole call, where it saved once per flag.
     - Changed: **`TrySetFieldConfiguration` returns every refusal; it no longer throws for some.**
       A negative `Weight` or `BM25k1`, a `BM25b` outside 0 to 1, `Filterable` or `Sortable` on a
       field with no type, and a vector field given another role were thrown as
       `ArgumentException` or `ArgumentOutOfRangeException`, where an unknown field name came back
       as `false`. All of them are now `false` with the reason in `error`, and nothing is applied.
       Code that caught the exception must test the return value instead. The obsolete
       `SetFieldConfiguration` and the setters on `Field` throw as before.
     - Changed: **a `Load` refuses a vector that is not numbers.** An empty array, an array with a
       string in it, or a value that is no array in a vector field fails the load with the
       document's key and the reason, as a vector of the wrong length already did and as Insert and
       Update do. Such a document used to be loaded with no vector and nothing said. A field that
       is absent or `null` is still a document with no vector.
     - Added: **a reason for the last `false`s without one.** `DeleteJsonRecord`,
       `DeleteJsonRecords` and `DeleteFilter` gain an overload with `out string error`, and so does
       `LoadDocumentFieldsFromDb` on `IServerSearchEngine`. A `false` from `DeleteJsonRecord` could
       be no such key, an engine that is not Ready, or an index build under way.
       `LoadDocumentFieldsFromDb` without a `Persistence` returns `false` with that reason, where it
       threw `NullReferenceException`. The existing signatures are unchanged and delegate.
     - Added: **`TryValidateFieldConfiguration(fields, out string error)`**: whether
       `TrySetFieldConfiguration` would accept the same argument, with nothing changed. For a caller
       that applies the change later or elsewhere, as a server that builds it into a clone.
     - Fixed: **two vector fields at the end of a record, with `NotKept`, threw on load**
       (`ArgumentOutOfRangeException` from the document store). Each was cut out with the comma
       beside it, and for the last two properties of a record that was the same comma.
     - Fixed: **`IEmbeddingIndex.TryGetVector` returns the vector at the length it had.** With the
       graph on 16-bit integers it came back a few millionths longer or shorter, and a vector read
       out and stored again moved by that much each time.
     - Fixed: **documents with one and the same vector no longer hide the others.** The same text
       embedded for many products gave many identical vectors, which closed into a group the search
       did not get out of: with a hundred identical vectors beside six distinct ones, none of the
       six was found. The graph now holds each distinct vector once, shared by the documents that
       have it, and a search returns all of them at the same score.
     - Added: **a vector graph is rebuilt by itself when removed vectors have gathered in it**, in
       the background beside searches and updates. `EmbeddingSetup.AutomaticRebuild` (on by default),
       `RebuildOwnerlessShare` (0.5: half as many removed as are left) and `RebuildOwnerlessMinimum`
       (1 000). With `AutomaticRebuild` off the same thresholds decide when
       `IEmbeddingIndex.RebuildDue` turns true, and the client calls `Index()` when it suits.
     - Added: **`SearchEngine.SaveEmbeddings` and `LoadEmbeddings`**: the search graphs of the
       vector fields written to a file, and taken from it at the next `Load` in place of being
       built. Call `LoadEmbeddings` after the field configuration is set and before `Load`. The file
       need not match the documents: the graph is brought in line with the vectors that are loaded,
       and is passed over and built as usual when it differs too much or was saved with another
       `GraphVectors`. `IEmbeddingIndex.RestoredFromSaved` says which happened. On 43 000 products
       with vectors of 1 536 numbers the `Load` took 25 s restored against 41 s built.
     - Added: **`IEmbeddingIndex.TryGetVector(documentKey, out vector)`**: the vector the search
       graph holds for a document, by a direct lookup, at the length it arrived with. It is the
       graph's copy, so near the numbers that arrived and not always the same: with the graph on
       16-bit integers each number is within 0.000015 of the vector's length of what arrived.
     - Changed: **an update keeps a vector's place, and a filter gets a full list.** A document whose
       vector is unchanged costs the graph nothing; a changed vector replaces the old one in one step
       (a search beside an update used to miss the document), and a deleted one's place is taken by
       the next vector added. A filtered vector search returned documents outside the filter, and
       dropped ones inside it, once any document lacked a vector or had been updated; it now returns
       as many as were asked for among the documents the filter admits, and measures them one by one
       when the filter admits few. `IEmbeddingIndex.OwnerlessCount` says how many removed vectors
       the graph still holds; `Index()` takes them out.
     - Fixed: **the library reports its own version.** The `Indx.dll` in 5.0.1 said 5.0.0 in its file
       and product version, so anything showing the version, such as the Indx server's Admin page,
       read 5.0.0.