iTool.MySearch 4.0.8

iTool.MySearch

iTool.MySearch owns the search contracts, Orleans grains, Lucene projection, query admission, durable source mutation path, and index recovery logic for a named search provider.

The public search contracts live under Contracts/ and use the iTool.MySearch namespace. The module no longer references iTool.ShareLibrary.Providers; product code registers the module with AddMySearchProvider and calls the contracts or Orleans grains. Lucene.Net.Store.Storage remains the lower level storage capability and does not own search policy.

Search indexes are projections. In Projection mode, the product database remains the source of truth. Durable mode stores the source document and mutation journal beside the index so a Silo can rebuild a damaged local generation. Both modes keep provider, partition, index, source revision, and index generation in the module boundary.

silo.AddMySearchProvider("documents", options =>
{
    options.StorageMode = MySearchStorageMode.Projection;
    options.MaxConcurrentQueriesPerSilo = 4;
});

The host supplies Orleans, Redis, identity options, and the lower level Lucene directory provider. The module does not create a second product database or expose Lucene internals to callers.

Disposable Lucene reader and writer mirrors use bounded process memory by default. This keeps a Runtime host independent of a writable node filesystem; Redis remains authoritative. A deployment that explicitly selects MySearchLocalCacheMode.FileSystem must also set LocalCacheRoot and owns that node-local cache lifecycle. MaximumInMemoryCacheBytes bounds each memory mirror.

Workload-specific acceptance

There is no universal 500 ms latency gate. Record dataset size, query shape, concurrency, source/result payload, topology, storage, CPU, memory, errors, overload behavior and recovery. The E2E runner only enforces P95 when ITOOL_SEARCH_P95_TARGET_MS is explicitly set to a positive finite number. Otherwise a functional success is reported as measured, with performance still requiring workload/resource comparison. Historical 500 ms failures retain their original reports but are not current release criteria. Elasticsearch reference runs must disclose analyzer, query semantics, durability, replicas and resource differences.

Filesystem checkpoint writes request preallocation before reading the source. This lets supporting filesystems reject insufficient physical space early; it is not a replacement for a bounded volume. The real Linux tmpfs fixture covers ENOSPC, preservation of a prior checkpoint, interrupted streams, process kill, abandoned allocation cleanup and an unrelated volume remaining writable. It does not establish production filesystem or power-loss safety.

Capacity-protected durable providers

These options are opt-in and do not change existing stored collections silently. Provision a bounded noeviction Redis instance first. A different Redis database number does not isolate memory, eviction, or server stalls. RequireDedicatedBackend compares the provider's actual Redis instance identity with the host's primary Redis and rejects a shared instance, including another DB.

services.AddMySearchRedisConnection("operational-events", isolatedRedisConfiguration);
services.AddMySearchProvider("operational-events", options =>
{
    options.StorageMode = MySearchStorageMode.Durable;
    options.Capacity.Enabled = true;
    options.Capacity.RequireDedicatedBackend = true;
    options.Capacity.SourceBucketCount = 4096;
    options.AdaptiveSharding.Enabled = true;
    options.LocalCacheMode = MySearchLocalCacheMode.FileSystem;
    options.LocalCacheRoot = nodeLocalCachePath;
    options.UseMemoryMappedFiles = true;
    options.CheckpointStoragePath = sharedCheckpointPath;
    // Only for explicitly disposable, newly created collections:
    options.DocumentRetention = TimeSpan.FromDays(7);
    options.RetentionInterval = TimeSpan.FromSeconds(1);
});

The engine atomically checks source-byte budgets at collection, tenant, provider, and source-bucket scope. Redis used_memory/maxmemory governs total backend admission, including all Redis DBs and index/checkpoint blobs. Unknown capacity or an eviction policy other than noeviction rejects growth. The default source and maintenance watermarks are 75% and 90%; maintenance headroom is not ordinary write capacity. Source quotas count serialized source bytes, not total physical allocator usage. Immutable file writes have a separate size limit and admission check. Stable mysearch_*_capacity_* errors are backpressure, not permission to discard durable business data or retry in a tight loop.

New protected collections persist their source-bucket layout. Existing unbucketed collections retain their original layout and are accounted once before admission. Changing SourceBucketCount does not remap existing data. A larger existing collection requires an explicit migration, not a configuration flip. Physical index splits use both measured load and TargetShardStorageBytes; memory mirrors cap the effective target at half their byte limit. The existing shard-count and migration budgets still apply, so reaching the maximum shard count is reported as a capacity limit.

Retention is based on first ingestion time, not last update time. It uses bounded conditional deletes, the existing source journal and checkpoint floor, and a persistent registry to revisit idle collections after restart. An empty collection rotates both recovery slots before unregistering its cleanup work. Revision tombstones are removed only in this explicitly disposable mode; conditional creation with revision zero also requires a source revision fence. Changing retention mode on an existing collection is rejected. Idempotency history remains bounded by IdempotencyHistoryCapacity and is not erased merely because the collection became empty.

Checkpoint replacement records pending cleanup in the same Redis transaction as catalog publication. Cleanup failures retain that record; subsequent maintenance retries before the unchanged-checkpoint fast path. Shared blobs still referenced by either live slot are retained. Redis file retirement is recorded atomically with metadata removal, and checkpoint pins prevent expiration. Unpublished Redis outputs expire; successful publication persists them.

Protected external checkpoints persist the complete upload plan and perform one Redis WAITAOF barrier before writing any external blob. Redis AOF and WAITAOF support (Redis 7.2+) are required; unavailable or unconfirmed intent durability rejects the upload. Catalog publication removes its upload intents atomically. Interrupted uploads are deleted in bounded batches on later maintenance, including abandoned filesystem temporary outputs. Maintenance registration is persistent for ordinary durable collections as well as expiring collections; cleanup does not expire ordinary business data.

External backends used with capacity protection must implement IMySearchCheckpointMaintenanceStore. Its exclusive ownership must cover in-flight writes until they end, using storage-enforced locking or fencing. An expiring application-only lock is insufficient. The filesystem implementation uses a shared, stable OS lock file per collection; verify cross-host locking on the deployment filesystem. Upgrade all collection owners together before enabling this protocol. Pre-upgrade external orphans which have no intent record still require a one-time storage reconciliation.

RAM mirrors, writer buffers, and admitted query work share a process-wide reservation budget. Deleted RAM files remain charged while reader clones reference them. This is an engine resource reservation, not an OS RSS limit. Filesystem mirrors and external checkpoint storage need dedicated bounded volumes/storage quotas and host monitoring. The backend implementation supplied through IMySearchCheckpointBlobStore remains responsible for physical storage quotas and access isolation.

Large-data exchange

Recovery reads the requested journal revisions in bounded pages and pipelines source reads only for the changed IDs. A physical split reads the selected shard's snapshot and catches up from the journal; it does not scan the whole logical source collection. Recovery keeps healthy physical shards.

StoreSourceByField enables independently selectable root source fields. Upgrade every reader before enabling it; old whole-source documents remain readable. Small stored values precede large bodies and the projection visitor stops after satisfying the requested fields. Lucene compression-block reads still impose overhead. Full-source queries necessarily return the requested document bodies. MaximumQueryResultBytes bounds hit responses, aggregation key state, and the serialized response.

UseMemoryMappedFiles uses Lucene's existing memory-mapped reader for filesystem mirrors. It avoids repeated file-stream seeks while reading sorting/doc-value data, but does not eliminate the initial mirror download. Cold mirror creation and checkpoint checksum validation still transfer more data than one query result. There is no claim of globally minimal I/O or a universal latency bound.

GetHealthAsync exposes protection status, source/journal bytes and limits, the persisted bucket count, effective split target, and process resource reservations. Distinguish these counters from Redis RSS, external checkpoint bytes, and node-local disk usage.

Local acceptance evidence — 2026-10-28/29 candidate

This is an unpublished working-tree candidate based on 6c4cf18bbaa6f938dda206d44b16151369012ac6, not the immutable public 4.0.4 package even though local assemblies still carry that version. Evidence is under outputs/idb-engine-20261028. The full ingestion run, mmap comparison, and final recovery runs used successive candidate builds; the evidence manifest records their binary hashes. These are local loopback tests on a 16-core Arm64 Mac with 128 GiB RAM, .NET 10.0.7, Redis 7.4.11, and six Silo processes.

Correctness passing does not mean latency passing. The harness records latency separately. Journal tail exchange is bounded by requested revisions instead of total history; projections still read compression blocks. Exact full-match counts, cold mirror acquisition, and checkpoint integrity scans have different data requirements. The evidence does not establish globally minimal or optimal I/O.

Outstanding deployment/design boundaries: no package publication or Workspace integration; source quotas are not per-tenant physical-byte quotas; engine reservations are not an OS memory/CPU sandbox; external disk capacity and cross-host maintenance ownership require backend enforcement; source-bucket resharding requires explicit migration; cross-slot Redis Cluster operation is not added by this change.

Redis client pins are coordinated at 3.4.20 across Foundation consumers. The transport regression corresponds to the upstream buffer-boundary fix. The deterministic encoder defect is established; the single earlier native crash's exact corruption chain was not reconstructed from a heap dump, so it is not asserted to have a proven unique cause.

2026-10-29 排序读取采样补充（验收未通过）

默认稳定 ID 排序沿用 Lucene 的字符串比较器，缓存每段不可变排序序号，不缓存完整查询结果。MaximumSortOrdinalCacheBytes 默认 64 MiB 进程上限（多个 provider 采用较小值），同时计入原进程内存预算；零关闭，容量不足回退原读取，最后一个段 core 读者关闭后释放。字典查找使用每个查询自己的 Lucene 读游标，避免跨线程共享游标。两项真实 Lucene 测试覆盖不同查询/中文/缺失值/逆序/翻页及容量和生命周期。

同一百万数据 200 并发首轮 1,000 请求：387.8 次/秒、P95 1,110.9 ms、最大 1,484.8 ms，仍未达到逐请求 <500 ms；这不是验收通过。六节点冷镜像 4.23 GB/17.7 秒仍需要继续减少。E2E 并发场景现把延迟达到预算、查询错误和不足三个查询节点均记为 failed 并以非零退出，单独统计各节点请求数与查询种类。

无采样补测：同一百万数据，20,000 请求/200 并发，0 查询错误，698.5 次/秒、P95 868.8 ms、最大 1,667.3 ms、4,215 次达到 500 ms；Lucene 执行 P95 93 ms。后续取消缓存热路径重复加锁，723.6 次/秒、P95 687.7 ms，仍失败，主要等待在服务内准入。每节点 2/4/8 执行槽位对照都失败，默认保持 4；没有降低客户端并发。冷镜像仍约 17 秒/4.23 GB。带采样运行包含 2,249 次过载拒绝，仅作为诊断。

可选按需写入工作目录（本地候选，默认关闭）

UseOnDemandWriterCache 让历史文件复用查询的不可变快照和块缓存，新文件继续本地生成及按原协议发布。与 UseOnDemandQueryCache 独立开关，共用 MaximumQueryBlockCacheBytes 及进程预算。仅支持 R1/R2 文件身份；RAM 本地容量仅计新文件，合并仍需要足够的新文件空间。快照在 IndexWriter 取得原写锁之后加载，避免交接期间覆盖前任的新提交；完整复制模式也采用此顺序。

128 条大正文索引约 31.8 MB，打开、单条更新、提交读取约 638 KB 历史内容，RAM/FS 两种模式的回滚和强制合并验证通过。百万六节点、查询均按需、同构建写者对照：冷启动加查询 Redis 输出约 840 MB → 390 MB，本地镜像约 604 MB → 0；这是特定夹具的实测，不表示所有写入或合并都无需读取历史。

完整适配层 overlay2 候选 252 通过、3 PostgreSQL 环境跳过；overlay4 延迟读缓冲记账后为 253 通过、3 环境跳过。其 20,000 请求/200 并发仍有 11 次过载、P95 716.7 ms；overlay2 混合读写出现进程预算拒绝，修复未读游标提前计缓冲后，overlay4 短期混合读写/查询节点故障验证通过（48,338 操作、0 错误）；百万损坏修复 10.10 秒通过。overlay4 的 200 并发门禁仍失败（P95 864.79 ms、8 次过载）。预约池并非 GC 堆/RSS 硬上限，实测托管堆曾到 2.49 GB；生产硬隔离、长时间验收未完成，未发布或在产品启用。

可选按需查询块缓存（本地候选，默认关闭）

设置 UseOnDemandQueryCache = true 后，查询冻结 R1/R2 文件身份，按实际读取加载 64 KiB 的 Redis 文件块；不复制整份查询镜像。MaximumQueryBlockCacheBytes 默认进程 256 MiB，并复用进程预算计入解码临时空间、块元数据和输入缓冲。LRU 合并并发读取、只淘汰无在途使用者的块；内存压力先回收空闲块。该选项不改变写者工作副本、持久文件格式或业务保留期，旧文件格式需先迁移。

旧代文件回收后若缺少所需块，整个查询失败并从新快照重试，不拼接新旧数据。真实 Redis 已验证旧代淘汰+缓存清空后正确读到新代正文、数据库隔离、并发同块只取一次、额度/LRU、内存压力让出缓存、在途 Dispose 保留预算直到复制完成。

百万六节点同构建对照：查询准备阶段 Redis 输出从 3.668 GB 降到 0.132 GB（约 -96.4%），准备从 17.50 秒降至 0.82 秒，后续运行出现 2.27 秒；写者仍有约 604 MB 本地工作副本且启动需要约 0.7–0.9 GB Redis 输出。128 条大正文的冷小字段查询：索引总文件 31.85 MB，实际 Redis 输出约 0.85 MB（包括元数据与 INFO 开销）。这些是具体验证负载的结果，不是所有查询的最优保证。

最新按需持续 20,000 请求/200 并发为 740.6 成功次/秒、P95 954.7 ms、68 次过载拒绝，期间 Redis 输出约 178 MB，仍然未通过延迟门禁。默认继续关闭；尚需完整最终版本故障、持续混合读写与发布验收。

首次打开失败与 RAM 预算回收

多分片首次打开中途失败会释放已打开镜像并清理当次部分文件，条件移除失败 Lazy 条目，后续修复存储后可以重新读取。Lucene 可遗弃未 Dispose 的 RAM input clone；预算在所有引用不可达并被 GC 回收后兜底释放，显式关闭仍及时归还，活跃副本不提前退款。旧实现文件遗留、异常缓存及 GC 后计数泄漏均已复现；新实现文件/内存两种模式 4 案例及完整适配层 242 项通过，3 项 PostgreSQL 环境跳过。百万损坏恢复再次通过；这不改变未完成的整体延迟和冷数据交换验收状态。

2026-10-29 按集合保留期与持久磁盘补验（未发布候选）

MySearchOptions.DocumentRetentionByIndex 以区分大小写的完整逻辑索引名覆盖该 provider 的 DocumentRetention，适用于同名索引的各 partition；字典值 null 表示该索引不自动过期，未配置的索引沿用全局值。保留期从首次入库时间计算，更新不延期；任何启用保留期的配置仍要求 Durable 和 Capacity.Enabled，正值至少一秒。机制继续由原维护登记、后台维护和有界删除执行，不在业务层扫描或维护第二套 TTL。

原无 age 元数据的非空集合不能直接开启过期：原持久化保护返回 mysearch_retention_policy_migration_required，必须显式迁移；不能伪造历史入库时间或静默删除。由启用变为禁用同样受持久化模式保护。调整已经启用的保留时长不改变业务正文。部署前应核对旧集合状态。

真实六 Silo 测试覆盖两个 partition 的 7 天、90 天与普通集合。写入后停止全部节点，仅将测试 age 元数据设为 8 天前，再以相同 ServiceId、新 ClusterId 重启；没有前台激活时 7 天集合归零，其余各 32 条完整保留，16.93 秒完成。这是时间元数据故障夹具，不是实际等待八天。

完整适配层最新 259 通过、3 PostgreSQL 环境跳过；首次完整运行的共享预算基线断言失败保留，测试现先完成其他用例遗留对象的终结，再测未读游标的增量预留，不在被测分配和释放之间强制 GC。私有候选包 4.0.5-idb.20261029.2 共 20 个包通过打包核验，仅用于本地集成，没有覆盖公开 4.0.4。

CheckpointCapacityAcceptance 增加独立小容量 ext4 夹具。32 MiB 镜像有效容量 27,234,304 字节；40 MiB 上传在读源之前失败，旧 8 MiB 检查点保持完整，截断输入释放预分配。子进程 SIGKILL 后维护删除归还空间，再写成功；卸载、重新挂载后全部文件 SHA-256 一致。它验证本机 Linux 持久文件系统的容量和进程故障，不等于断电、跨主机共享锁或生产挂载验证。

当前集成分支已由其他会话推进至 c61b03d389225087982e3b99097bbb9ecd578ac5（声明 4.0.5）；本任务未发布。ext4 与十万条提交策略对照使用此基线加本地候选，既有私有 r2 和千万运行使用更早的独立候选，不可混为同一版本验收。后续发布必须选择新的未占用版本并重新固定来源。

Activation fault boundary (2026-10-29): writer and query OnActivateAsync now reuse MySearchInfrastructureFailure and the existing serializable MySearchUnavailableException. Real isolated Redis pause (10 seconds, provider timeout 3 seconds) reproduced CodecNotFoundException on the previous private r2 writer activation; private 4.0.6-idb.20261029.1 returns the exact transient contract, keeps the primary Redis available, and preserves source/query results after recovery. Eight product policy/fault tests and 259 adapter tests passed (3 PostgreSQL environment skips). Original unexplained background write failures remain open. No public publication or deployment.

Failed activation cleanup

Writer activation owns every opened writer and migration worker even when a later checkpoint or journal operation fails. Failure releases timers and rolls writers back to their last commit through the existing disposal path, returning memory reservations and write locks without deleting committed source data. Normal deactivation still flushes pending work before disposal. The real checkpoint-I/O fault regression reproduced three leaked reservations on the previous private candidate and passed with the cleanup fix; final coordinated package acceptance is still pending.

Active batch memory

The shared process pool holds writer-buffer reservations for the lifetime of a writer. Maximum mutation payload allowances are held only during active mutation/replay work, and are returned on success or exception. A small mutation reserves against its document count and document-size ceiling, capped by the batch ceiling. Split scans account for their pending batch plus one source document. These conservative reservations are admission accounting, not a guarantee on total managed heap or RSS; process/container hard limits still apply. Private r4 adapter and diagnostic product regressions passed, with final package acceptance pending.