IndxSearchLib 5.0.1

• net10.0

  • CommunityToolkit.HighPerformance (>= 8.4.0)
  • HNSW (>= 26.6.1848)
  • MessagePack (>= 3.1.7)
  • Microsoft.Data.Sqlite (>= 10.0.3)
  • Microsoft.Extensions.Logging.Abstractions (>= 10.0.3)
  • SQLitePCLRaw.bundle_e_sqlite3 (>= 3.0.3)

Three fixes to 5.0.0, and what is known to be wrong or weak in 5.0 after them. The first two
     fixes are of answers that were wrong with nothing said, so take this release if you load
     vectors or rely on a failed load being a failed load.
     
     - Fixed: **a filter on a vector search returned the wrong documents when some documents had no
       vector.** The filter was asked about the vector's number in the search graph, where it is
       numbered by the document's place in the store, so every document without a vector shifted
       all that followed: a filter on red returned blue documents and dropped red ones, with no
       error.
     - Fixed: **a `Load` that fails leaves no documents, and `Index()` no longer builds on one.**
       The load reported the failure and set `Error`, but kept the documents it had read, and
       `Index()` then succeeded: the engine was `Ready` on part of the data. `Index()` is now
       refused in `Loading`, `Indexing` and `Error`, with the reason on the monitor.
     - Fixed: **a field made `Facetable` while a search was running could stay without facets.**
       The lists of searchable and facetable fields are cached, and a search that was building one
       as a role was set stored its list after the change had cleared the cache. A list built
       across a change is no longer kept.
     - Changed: **a vector field has no other role, and a wrong vector is refused.**
       `TrySetFieldConfiguration` throws `ArgumentException` for `Embeddable` together with
       Searchable, Filterable, Facetable, Sortable, WordIndexing or PreloadFilters, or on a field
       that is not an array of numbers, and `Load` refuses such a configuration. A configuration
       saved with such a combination is repaired when it is read: the vector field keeps
       `Embeddable` and loses the other roles, which never did anything useful on a vector.
       `InsertJsonRecord(s)`, `UpdateJsonRecord(s)` and `UpdateField` return `false`, storing
       nothing, for a vector that is empty, of another length than the field's, or not numbers;
       they used to store the document and then refuse, or take it in without a vector.
     
     ### Known limitations of 5.0
     
     Vector search:
     
     - **A dataset with a vector field takes no insert, update or delete.** Every update of a
       document removes its vector and adds it again, also when only another field changed (a
       price), and after a few the vector search no longer finds the document; `Index()` does not
       repair it. A delete leaves a result list short. Load the dataset, index it and search it; to
       change it, load it again.
     - **Documents with one and the same vector hide the others.** The same text embedded for many
       products gives many identical vectors, which close into a group the search does not get out
       of.
     - **A vector search misses some of the true nearest, more on a larger catalogue.** Measured on
       real embeddings of 1 536 numbers against the exact answer: of the ten nearest it found 73 %
       on 43 000 products and 96 % on 29 000. The order among those it does find is right.
     - **The first vector decides the length for the field.** If the first document's vector has
       the wrong length, every correct one after it is refused.
     - **A `Load` takes in a document whose vector is empty or holds something other than
       numbers**, with no vector and no word. Insert and update refuse it.
     - **A vector is taken to be of unit length.** The score is the dot product as it is, so
       vectors of other lengths are not ranked by cosine.
     - **A document with a vector costs about 110 KB of memory** for a vector of 1 536 numbers.
     
     In general:
     
     - **`Status` can report `Ready` a few milliseconds before `Index()` has finished**, and a
       search in that moment is answered "not now" (`Result.DidTimeOut`). Waiting on the
       `ProcessMonitor` instead of polling the state avoids it.
     - **A role or a weight set straight on a `Field`** is not saved correctly by an engine with a
       `Persistence`: `Searchable`, `Sortable` and `Weight` are saved with the value from before,
       and the other roles are not saved at all. `TrySetFieldConfiguration` followed by the
       caller's own save is not affected.
     - **`TrySetFieldConfiguration` throws for a value or a role it refuses** (a negative weight, a
       `BM25b` outside 0 to 1, `Filterable` on a field with no type, a vector field with another
       role) and returns `false` only for a field name it does not have.
     - **`DeleteJsonRecord`, `DeleteJsonRecords`, `DeleteFilter` and `LoadDocumentFieldsFromDb`
       return `false` without a reason.** For `DeleteJsonRecord` that is no document with the key,
       an engine that is not `Ready`, or an index build under way.
     
     All of these are addressed in the release that follows 5.0.