AgentEval 1.0.0-rc.1

Prefix Reserved
.NET 8.0 This package targets .NET 8.0. The package is compatible with this framework or higher.
This is a prerelease version of AgentEval. 
dotnet add package AgentEval --version 1.0.0-rc.1
                    


                    

                
NuGet\Install-Package AgentEval -Version 1.0.0-rc.1
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="AgentEval" Version="1.0.0-rc.1" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="AgentEval" Version="1.0.0-rc.1" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="AgentEval" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add AgentEval --version 1.0.0-rc.1
                    


                    

                
#r "nuget: AgentEval, 1.0.0-rc.1"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package AgentEval@1.0.0-rc.1
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=AgentEval&version=1.0.0-rc.1&prerelease
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=AgentEval&version=1.0.0-rc.1&prerelease
                    


                            Install as a Cake Tool

AgentEval

The .NET Evaluation Toolkit for AI Agents

Built first for Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI. What RAGAS and DeepEval do for Python, AgentEval does for .NET.

Release candidate: 1.0.0-rc.1. This prerelease is for feedback and can still change before 1.0.0. From 1.0.0, stable library APIs and documented CLI commands, options and exit codes follow semantic versioning; incompatible changes require a major version. Types marked [Experimental] and fronts labelled Beta or Experimental remain outside that promise: see the authoritative Feature Maturity table. API snapshots record the bundled library's current shape without promoting those exceptions. Changes are listed in the CHANGELOG. Do not use AgentEval in production or safety-critical systems without independent review, testing and hardening; see the disclaimer.

Upgrading from 0.43? See the 1.0 migration guide.

Features

  • 🎯 Tool Tracking β€” Monitor tool/function calls with timing, arguments, and ordering
  • βœ… Fluent Assertions β€” Expressive assertions with rich failure messages, because reasons, and assertion scopes
  • πŸ“Š Performance Metrics β€” TTFT, latency, tokens, cost estimation for 8+ models
  • πŸ”¬ RAG Metrics β€” Faithfulness, relevance, context precision/recall, answer correctness
  • πŸ›‘οΈ Red Team Security β€” 14 attack types, 264 probes, full OWASP LLM Top 10 coverage
  • πŸšͺ Gatekeeper β€” Fail-closed runtime enforcement: block forbidden/poisoned tool calls before they run, red-team probes as live guards, and human-in-the-loop approval
  • βš–οΈ Responsible AI β€” Toxicity, bias, and misinformation detection metrics
  • πŸ“ˆ Stochastic Evaluation β€” Statistical model comparison with multi-run analysis
  • πŸ”„ Trace Record & Replay β€” Deterministic CI testing without LLM calls
  • 🎯 Calibrated Judge β€” Several LLM judges score the same reply and vote, with their agreement reported
  • πŸ”Œ Extensible β€” Adapter pattern for any agent framework

Quick Start

using AgentEval.Assertions;
using AgentEval.Core;
using AgentEval.MAF;
using AgentEval.Models;

// Create evaluation harness (evaluatorClient: the IChatClient that grades replies)
var harness = new MAFEvaluationHarness(evaluatorClient);

// Wrap your Microsoft Agent Framework AIAgent for evaluation
var agent = new MAFAgentAdapter(aiAgent);

// Run evaluation with tool tracking. ModelName selects the price used to estimate cost;
// without a name found in the price table, no cost is estimated.
var result = await harness.RunEvaluationAsync(agent, new TestCase
{
    Name = "Feature Planning Test",
    Input = "Plan a user authentication feature",
    EvaluationCriteria = ["Should include security considerations"]
}, new EvaluationOptions { ModelName = "gpt-4o" });

// Assert tool usage with "because" reasons
result.ToolUsage!
    .Should()
    .HaveCalledTool("SecurityTool", because: "auth features require security review")
        .BeforeTool("FeatureTool")
        .WithoutError()
    .And()
    .HaveNoErrors();

// Assert performance. Cost needs observed usage and a configured price for the identified model.
// Unavailable measurements are reported as inconclusive.
result.Performance!
    .Should()
    .HaveTotalDurationUnder(TimeSpan.FromSeconds(10))
    .HaveEstimatedCostUnder(0.10m);

Red Team Security Scanning

using AgentEval.RedTeam;
using AgentEval.RedTeam.Reporting;

var result = await AttackPipeline.Create()
    .WithAllAttacks()
    .ScanAsync(agent);

result.Should()
    .HavePassed()            // fails on any compromised probe, and on a scan too inconclusive to trust
    .HaveMinimumScore(85);   // percentage of probes resisted

await new SarifReportExporter().ExportToFileAsync(result, "security-report.sarif");

Trace Record & Replay

Capture agent executions for deterministic replay β€” no LLM calls needed in CI:

using AgentEval.Tracing;

// Record
await using var recorder = new TraceRecordingAgent(realAgent, "weather_test");
var response = await recorder.InvokeAsync("What's the weather?");
await recorder.SaveAsync("trace.json");

// Replay (deterministic, free)
var trace = await TraceSerializer.LoadFromFileAsync("trace.json");
var replayer = new TraceReplayingAgent(trace);
var replayed = await replayer.InvokeAsync("What's the weather?");

Model Comparison

using AgentEval.Comparison;

var comparer = new ModelComparer(new StochasticRunner(harness));

// CreateAgent(deployment) is your code: it returns an IEvaluableAgent for that model
var results = await comparer.CompareModelsAsync(
    factories: new IAgentFactory[]
    {
        new DelegateAgentFactory("gpt-4o", "GPT-4o", () => CreateAgent("gpt-4o")),
        new DelegateAgentFactory("gpt-4o-mini", "GPT-4o Mini", () => CreateAgent("gpt-4o-mini"))
    },
    testCases: testSuite,
    options: new ModelComparisonOptions(RunsPerModel: 5));

Console.WriteLine(results.ToMarkdown());

The quality, speed, cost and reliability scores rank the models against each other (best 100, worst 0), and cost is priced from a single model name. See Model Comparison for how the scores are computed.

Quality Assurance

  • The test suite runs in CI on .NET 8, 9 and 10; the build status shows the latest result

Installation

dotnet add package AgentEval --prerelease

Single package, modular internals β€” the AgentEval package embeds these assemblies; none of them is published as a separate package:

  • AgentEval.Abstractions β€” Public contracts and interfaces
  • AgentEval.Core β€” Metrics, assertions, comparison, tracing
  • AgentEval.DataLoaders β€” Data loading and export (JSON, YAML, CSV, JSONL)
  • AgentEval.MAF β€” Microsoft Agent Framework integration, including Gatekeeper
  • AgentEval.Memory β€” Memory evaluation, benchmarks, LongMemEval, HTML reporting
  • AgentEval.RedTeam and AgentEval.RedTeam.Gatekeeper β€” Security testing, and red-team evaluators as runtime gates
  • AgentEval.Evals.Agentic and AgentEval.Evals.Performance β€” The agentic evaluator suite and the performance benchmarks
  • AgentEval.Compliance.Core, AgentEval.Compliance.Gdpr and AgentEval.Compliance.EuAiAct β€” Compliance benchmarks
  • AgentEval.Rendering.Pdf β€” PDF report rendering

The Copilot Studio add-on (AgentEval.MAF.CopilotStudio) is not part of this package and is not on NuGet; build it from the repository if you need it.

Service Registration

// Register all services at once (recommended):
services.AddAgentEvalAll();

// Or register selectively:
services.AddAgentEval();              // Core services only
services.AddAgentEvalDataLoaders();   // DataLoaders + Exporters
services.AddAgentEvalRedTeam();       // Red Team security testing
services.AddAgentEvalMemory();        // Memory evaluation (AgentEval.Memory.Extensions)
services.AddAgentEvalAgentic();       // Agentic evaluator options (AgentEval.Evals.Agentic)

Documentation

License

MIT License β€” See LICENSE for details.

Deterministic evals

A deterministic eval is one measurement of an agent run, computed in code β€” no model, no cost, same answer every time. Register one with AgentEvalBuilder.AddEval(eval, floor): the door takes the eval and the chance floor it is judged against, because a score you cannot compare to luck is not a measurement. See Deterministic evals for the contract β€” what the eval sees, what null versus [] tool calls mean, and how to say "this could not be measured" without saying "this scored zero".

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
1.0.0-rc.1 78 10/25/2026
0.43.0-beta 392 10/6/2026
0.42.0-beta 276 10/1/2026
0.41.0-beta 14,595 9/21/2026
0.40.0-beta 84 9/20/2026
0.39.0-beta 166 9/16/2026
0.38.0-beta 418 9/15/2026
0.37.0-beta 87 9/14/2026
0.36.0-beta 110 9/13/2026
0.35.0-beta 1,252 9/8/2026
0.34.0-beta 110 9/4/2026
0.33.0-beta 137 9/2/2026
0.32.0-beta 107 9/1/2026
0.31.0-beta 161 8/30/2026
0.30.0-beta 107 8/29/2026
0.29.0-beta 161 8/28/2026
0.28.0-beta 107 8/22/2026
0.27.0-beta 94 8/22/2026
0.26.0-beta 122 8/19/2026
0.25.0-beta 103 8/17/2026
Loading failed

Per-version release notes: https://github.com/AgentEvalHQ/AgentEval/blob/main/CHANGELOG.md