Bitzsoft.Integrations.AI.Testing
1.2.0
dotnet add package Bitzsoft.Integrations.AI.Testing --version 1.2.0
NuGet\Install-Package Bitzsoft.Integrations.AI.Testing -Version 1.2.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Bitzsoft.Integrations.AI.Testing" Version="1.2.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Bitzsoft.Integrations.AI.Testing" Version="1.2.0" />
<PackageReference Include="Bitzsoft.Integrations.AI.Testing" />
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Bitzsoft.Integrations.AI.Testing --version 1.2.0
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: Bitzsoft.Integrations.AI.Testing, 1.2.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Bitzsoft.Integrations.AI.Testing@1.2.0
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Bitzsoft.Integrations.AI.Testing&version=1.2.0
#tool nuget:?package=Bitzsoft.Integrations.AI.Testing&version=1.2.0
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
Bitzsoft.Integrations.AI.Testing
面向 AI、Agent 与 MCP 连接器下游宿主的独立测试支持包。聚合全部无测试框架锁定(不传递引入 xUnit / NUnit / MSTest)的存储端口契约测试基类、进程内测试替身、受控时钟以及基于 Microsoft.Extensions.AI.Evaluation 的 FDE 黄金数据集评估运行器。
安装与兼容性
dotnet add package Bitzsoft.Integrations.AI.Testing
- 目标框架:
net8.0;net10.0(全面开启<IsAotCompatible>true</IsAotCompatible>,零反射序列化); 1.2.0迁移与破坏性变更说明:自1.2.0起 MCP 契约测试基类(McpAccessTokenStoreContractTestsBase、McpCandidateStoreContractTestsBase)依赖已由Bitzsoft.Integrations.McpServer切换至Bitzsoft.Integrations.AI.Mcp。测试项目升级本包至1.2.0时请同步将旧包McpServer原子切换为AI.Mcp并更新using命名空间(详见 ADR-0015);- 仅面向宿主的
*.Tests测试项目安装,严禁被生产运行时程序集引用; - 核心生产包(
AI.Governance、AI.Conversation等)对本包保持零依赖。
契约测试基类列表(共 23 个)
| 基类 | 验证端口 / 运行期能力 | 核心不变量 |
|---|---|---|
AIPendingActionStoreContractTestsBase |
IAIPendingActionStore |
跨租户隔离、会话去重、乐观并发抢占、到期扫描、取消传播 |
AIUsageCounterStoreContractTestsBase |
IAIUsageCounterStore |
固定窗口累加、过期归零、键隔离、高并发增量、取消传播 |
AIRequestLedgerStoreContractTestsBase |
IAIRequestLedgerStore |
三元组幂等创建、乐观并发跃迁、心跳续期版本保护、会话租约互斥、取消传播 |
AIProviderCatalogStoreContractTestsBase |
IAIProviderCatalogStore |
启用过滤与排序、租户与工作区隔离、健康状态幂等写入、取消传播 |
AISkillDefinitionStoreContractTestsBase |
IAISkillDefinitionStore |
内置基准幂等同步、全局与租户覆盖隔离及删除恢复、取消传播 |
AIAgentDefinitionStoreContractTestsBase |
IAIAgentDefinitionStore |
内置智能体幂等同步、全局与租户覆盖隔离及删除恢复、取消传播 |
AIConversationSessionStoreContractTestsBase |
IAIConversationSessionStore |
租户/用户归属隔离、工作区绑定、并发版本保护、消息序号与修补、分叉与回合重放、取消传播 |
AIBackgroundJobStoreContractTestsBase |
IAIBackgroundJobStore |
幂等建单与租户隔离、优先级抢占与租约到期接管、CAS 跃迁与终态保护、取消传播 |
McpAccessTokenStoreContractTestsBase |
IMcpAccessTokenStore |
哈希一致签发、受控时钟过期与清理、本人/管理面吊销隔离、取消传播 |
McpCandidateStoreContractTestsBase |
IMcpCandidateStore |
候选批量/单条更新、黑名单增删查确定性排序、取消传播 |
MemoryStoreContractTestsBase |
IAIMemoryStore |
租户与用户物理隔离、双区块字符与条数容量硬上限、提示注入与密钥阻断、增改删幂等、取消传播 |
SkillResourceReaderContractTestsBase |
IAISkillResourceReader |
跨租户隔离、正式态与沙箱草稿态可见性隔离、路径穿越别名阻断、分页与安全围栏闭合、取消传播 |
ExecutionArtifactStoreContractTestsBase |
IExecutionArtifactStore |
三重隔离、分页切片、受控时钟 TTL 过期回收、会话级清理隔离、取消传播 |
ArtifactAnalysisEngineContractTestsBase |
IArtifactAnalysisEngine |
跨表连接、十进制精度保真、同环比除零保护、派生血缘追踪、跨租户拦截 |
TurnFeedbackStoreContractTestsBase |
IAITurnFeedbackStore |
租户/用户/请求幂等覆盖、跨租户与跨用户隔离、UTF-16 安全截断、取消传播 |
SkillDraftStoreContractTestsBase |
IAISkillDraftStore |
沙箱草稿态保存、租户隔离、状态流转与审核隔离、取消传播 |
ScheduledAgentTriggerStoreContractTestsBase |
IAIScheduledAgentTriggerStore |
租户/用户隔离、到期查询、幂等认领与冷却窗去重、乐观并发版本保护、取消传播 |
CrossProviderFailoverContractTestsBase |
OrderedFailoverChatClient / IAIFailoverHealthTracker / IAICrossProviderHistorySanitizer |
首片前丢弃脏缓冲切换、首片后绝不重放、跨厂商剥离加密签名与归一化工具调用 ID、租户健康隔离与半开防惊群 |
ModelRoleIsolationGuardContractTestsBase |
HermesRoleRoutedChatClient / AIModelExecutionRole / ITurnTriageRouter |
轻量角色(门控/压缩/内务/心跳)绝不误触主推理模型、未配置独立轻模型时零开销旁路 |
ProviderReasoningEffortContractTestsBase |
AIReasoningEffortPolicy / HermesProviderCapabilityRegistry |
自定义网关模型映射、关闭禁发参、单档位拒绝不误伤高阶能力、自动循环上限钳制、取消传播 |
ResponseVerifierPipelineContractTestsBase |
AIResponseVerificationHook / AIVerifierOptions / IAgentResponseVerifier |
五态核查处置模式、按核查器覆写、单次自纠回环、阻断安全兜底文案与最终裁决动作遥测 |
ToolSubgraphSelectorContractTestsBase |
IToolSubgraphSelector / IGlossaryExpander |
两跳桥接、词汇衰减展开、RBAC 前置优先红线、环图防死循环与预算硬顶、取消传播 |
UiComponentCatalogContractTestsBase |
IAiUiComponentCatalog / IPresentationGuard |
五类内置组件预注册、双重载自定义校验、不可绕过安全基线、数据引用多租户隔离、呈现守卫降级阈值与线程安全 |
测试替身与 FDE 黄金评估运行器
AITestFakeTimeProvider:可控TimeProvider测试时钟,支持确定性推进租约过期与限流时间窗口;AITestFakeChatClient:可脚本化队列、流式分片、TTFT 时钟快进、工具调用仿真及调用轨迹审计的IChatClient替身;AITestFakeCacheEpochCoordinator:支持可用性故障开关、单调纪元计数与 TTL 过期的IAICacheEpochCoordinator替身;AITestFakeOutboundRedactor:支持敏感词替换规则、跨境区域阻断策略及调用审计的IAIOutboundRedactor替身;AITestInMemoryProviderCatalogStore:内置多租户与工作区过滤及健康事件追踪的IAIProviderCatalogStore进程内参考实现;AITrajectoryReplayRunner/AITrajectoryBehavioralAssertions:基于脱敏 JSONL 轨迹执行锁定历史工具返回、零外部副作用的离线确定性重放与工具调用序列/缓存命中率/数字对账行为断言;AIGoldenEvaluationRunner/AIHermesTrajectoryEvaluator/AILlmRubricEvaluator/AIGoldenSuiteRegressionComparator:基于Microsoft.Extensions.AI.Evaluation10.11.0 组合IEvaluator/CompositeEvaluator,覆盖 8 项hermes.*指标、零推理快路裁判(解析失败返回Inconclusive)、FinOps 成本上限、出站脱敏与跨版本退化基线对比(ToMarkdownSummary())。
完整最小示例
下面演示结合 AITestFakeTimeProvider、AITestFakeChatClient、AITestFakeOutboundRedactor 与 AIGoldenEvaluationRunner 执行单条黄金评估验证:
using Bitzsoft.Integrations.AI.Abstractions;
using Bitzsoft.Integrations.AI.Testing;
using Microsoft.Extensions.AI;
var clock = new AITestFakeTimeProvider(new DateTimeOffset(2026, 11, 14, 0, 0, 0, TimeSpan.Zero));
var chatClient = new AITestFakeChatClient(clock)
.EnqueueTextResponse(
"已完成季度结算对账单关键指标合规核查。",
usage: new UsageDetails { InputTokenCount = 1_000, CachedInputTokenCount = 400, OutputTokenCount = 200 },
toolCalls: [new FunctionCallContent("call-1", "query_settlement_items")],
timeToFirstToken: TimeSpan.FromMilliseconds(120),
interTokenDelay: TimeSpan.FromMilliseconds(30));
var redactor = new AITestFakeOutboundRedactor()
.AddSensitiveTerm("440301199001011234", "[ID_REDACTED]");
var runner = new AIGoldenEvaluationRunner(new AIGoldenEvaluationOptions
{
TimeProvider = clock,
OutboundRedactor = redactor,
DefaultPricing = new AIFinOpsModelPricing(InputPricePerMillion: 2.0m, OutputPricePerMillion: 8.0m, CachedInputPricePerMillion: 0.5m),
});
AIGoldenCaseReport report = await runner.EvaluateCaseAsync(
chatClient,
new AIGoldenEvaluationCase
{
CaseId = "settlement-review-001",
Messages = [new ChatMessage(ChatRole.User, "请核验客户 440301199001011234 的季度结算对账单关键指标。")],
RequiredResponseSubstrings = ["关键指标"],
ForbiddenLeakedTerms = ["440301199001011234"],
MaxTimeToFirstTokenMs = 300,
MaxTotalLatencyMs = 800,
MaxEstimatedCost = 0.05m,
ExpectedToolCalls = ["query_settlement_items"],
});
Console.WriteLine($"Passed={report.Passed}, TTFT={report.TimeToFirstTokenMs}ms, Cost={report.EstimatedCost} {report.CurrencyCode}");
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 was computed. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net10.0
- Bitzsoft.Integrations.AI (>= 1.2.0)
- Bitzsoft.Integrations.AI.Abstractions (>= 1.2.0)
- Bitzsoft.Integrations.AI.Conversation (>= 1.2.0)
- Bitzsoft.Integrations.AI.Governance (>= 1.2.0)
- Bitzsoft.Integrations.AI.Hermes (>= 1.2.0)
- Bitzsoft.Integrations.AI.Mcp (>= 1.2.0)
- Bitzsoft.Integrations.AI.Skills (>= 1.2.0)
- Microsoft.Extensions.AI.Abstractions (>= 10.11.0)
- Microsoft.Extensions.AI.Evaluation (>= 10.11.0)
-
net8.0
- Bitzsoft.Integrations.AI (>= 1.2.0)
- Bitzsoft.Integrations.AI.Abstractions (>= 1.2.0)
- Bitzsoft.Integrations.AI.Conversation (>= 1.2.0)
- Bitzsoft.Integrations.AI.Governance (>= 1.2.0)
- Bitzsoft.Integrations.AI.Hermes (>= 1.2.0)
- Bitzsoft.Integrations.AI.Mcp (>= 1.2.0)
- Bitzsoft.Integrations.AI.Skills (>= 1.2.0)
- Microsoft.Extensions.AI.Abstractions (>= 10.11.0)
- Microsoft.Extensions.AI.Evaluation (>= 10.11.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.