ElBruno.BM25 0.5.0

ElBruno.BM25 — Lightweight BM25 Full-Text Search for .NET

Production-ready BM25 full-text search library with zero external dependencies. Index millions of documents, search in milliseconds, and integrate seamlessly into RAG pipelines, knowledge bases, and hybrid search systems.

🎯 Quick Start (5 minutes)

Installation

dotnet add package ElBruno.BM25

One-Minute Example

using ElBruno.BM25;

// 1. Prepare your documents
var documents = new[]
{
    new { Id = 1, Title = "Machine Learning Basics", Content = "Learn ML fundamentals" },
    new { Id = 2, Title = "Deep Learning Guide", Content = "Neural networks and deep learning" },
    new { Id = 3, Title = "NLP Fundamentals", Content = "Natural language processing basics" }
};

// 2. Create an index
var index = new Bm25Index<dynamic>(
    documents,
    doc => doc.Content  // Extract searchable text
);

// 3. Search
var results = index.Search("learning", topK: 10);

// 4. Display results
foreach (var (doc, score) in results)
{
    Console.WriteLine($"{doc.Title}: {score:F2}");
}

Output:

Machine Learning Basics: 2.45
Deep Learning Guide: 1.89
NLP Fundamentals: 0.56

✨ Features

Core

• ✅ Lightweight — ~200 lines core algorithm, zero external dependencies
• ✅ Fast — Index 1M documents in <5s, search in <50ms
• ✅ Production-ready — Fully tested with 70+ unit tests
• ✅ Thread-safe — Safe for concurrent reads
• ✅ .NET 8.0+ — Modern async/await patterns

Search & Indexing

• 📄 Dynamic indexing — Add/remove documents on the fly
• 🔍 Flexible search — topK results, threshold filtering, batch search
• 📊 Score explanation — Debug why a document scored high
• 🎯 Custom queries — Cancellation tokens for long-running searches

Advanced

• 🎛️ Parameter tuning — Automatic grid search optimization (K1, B, Delta)
• 📝 Custom tokenizers — Simple, English (Porter stemming), or your own
• 💾 Persistence — Save/load indexes to disk in JSON format
• 📈 Batch operations — Search multiple queries efficiently
• 📊 Index statistics — Document count, term frequency, vocabulary richness

📖 Usage Examples

Basic Search

var index = new Bm25Index<Article>(
    articles,
    article => article.Content
);

var results = index.Search("machine learning", topK: 5);

English Tokenizer (with Porter Stemming)

using ElBruno.BM25.Tokenizers;

var index = new Bm25Index<Article>(
    articles,
    article => article.Content,
    tokenizer: new EnglishTokenizer()  // Stems: "running" → "run", "authentication" → "authent"
);

Custom Tokenizer

var customTokenizer = new CustomTokenizer(text => 
{
    // Your domain-specific logic here
    return text.ToLower().Split(' ').ToList();
});

var index = new Bm25Index<Article>(articles, a => a.Content, customTokenizer);

Parameter Tuning

var tuner = new Bm25Tuner<Article>(index);
var validationQueries = new List<(string query, List<Article> relevant)>
{
    ("machine learning", relevantArticles1),
    ("neural networks", relevantArticles2)
};

var optimizedParams = await tuner.TuneAsync(validationQueries, TuningMetric.F1);
index.Parameters = optimizedParams;

Score Explanation

var query = "machine learning";
var doc = articles[0];

// Simple explanation as dictionary
var explanation = index.ExplainScore(doc, query);
Console.WriteLine($"Total Score: {explanation["total_score"]}");

// Detailed breakdown
var detailed = index.ExplainScoreDetailed(doc, query);
Console.WriteLine($"Matched Terms: {detailed.MatchedTermCount}");
foreach (var term in detailed.TermScores)
{
    Console.WriteLine($"  {term.Key}: IDF={detailed.TermIDFs[term.Key]:F2}, Score={term.Value:F2}");
}

Batch Search

var queries = new[] { "machine learning", "neural networks", "NLP" };
var batchResults = await index.SearchBatch(queries, topK: 5);

foreach (var (query, results) in batchResults)
{
    Console.WriteLine($"\nQuery: {query}");
    foreach (var (doc, score) in results)
    {
        Console.WriteLine($"  {doc.Title}: {score:F2}");
    }
}

Persistence

// Save index to disk
index.SaveIndex("my_index.json");

// Load it back later
var restoredIndex = Bm25Index<Article>.LoadIndex("my_index.json");
var results = restoredIndex.Search("machine learning");

Dynamic Indexing

var index = new Bm25Index<Article>(articles, a => a.Content);

// Add new document
var newArticle = new Article { Title = "New ML Article", Content = "..." };
index.AddDocument(newArticle);

// Remove document
index.RemoveDocument(oldArticle);

// Reindex entire collection
index.Reindex(updatedArticles);

⚡ Performance

Memory Usage:

• 100K documents: ~50-100 MB
• 1M documents: ~300-500 MB
• Depends on document size and vocabulary

🔧 BM25 Algorithm

ElBruno.BM25 implements the BM25F (Best Matching 25 with Fields) formula, a proven ranking function in information retrieval.

Score Formula:

BM25(q,d) = Σ IDF(q_i) * ((k1 + 1) * TF(q_i,d)) / (TF(q_i,d) + k1(1 - b + b * |d|/avgdl))

Parameters:

• k1 (1.5) — Controls term frequency saturation. Higher = more impact from repeated terms.
• b (0.75) — Document length normalization. 0 = no normalization, 1 = full normalization.
• delta (0.5) — Smoothing factor for IDF calculation.

Preset Parameters:

• Bm25Parameters.Default — Balanced (k1=1.5, b=0.75)
• Bm25Parameters.Aggressive — For large corpora (k1=2.0, b=1.0)
• Bm25Parameters.Conservative — For small corpora (k1=1.0, b=0.5)

📚 API Reference

Bm25Index<T>

Constructor:

new Bm25Index<T>(
    IEnumerable<T> documents,
    Func<T, string> contentSelector,
    ITokenizer? tokenizer = null,                    // Defaults to SimpleTokenizer
    Bm25Parameters? parameters = null,               // Defaults to Default
    bool caseInsensitive = true
)

Key Methods:

Properties:

• DocumentCount — Number of indexed documents
• TermCount — Number of unique terms
• Parameters — Get/set BM25 parameters

Tokenizers

ITokenizer Interface:

public interface ITokenizer
{
    List<string> Tokenize(string text);    // Convert text to terms
    string Normalize(string term);         // Normalize single term
    string Name { get; }                   // Tokenizer name
}

Built-in Tokenizers:

• SimpleTokenizer — Whitespace split, lowercase, no stemming
• EnglishTokenizer — Includes Porter stemming for English
• CustomTokenizer — User-defined function

Parameter Tuning

var tuner = new Bm25Tuner<T>(index);
var optimized = await tuner.TuneAsync(
    validationQueries,                     // (query, relevantDocs) tuples
    metric: TuningMetric.F1,               // Metric to optimize
    ct: cancellationToken
);

TuningMetric Options:

• Precision — % of retrieved docs that are relevant
• Recall — % of relevant docs that are retrieved
• F1 — Harmonic mean (recommended for balanced tuning)
• NDCG — Ranking quality

🤝 Integration Examples

RAG Pipeline

// 1. Index knowledge base
var kb = LoadKnowledgeBase();
var index = new Bm25Index<KbArticle>(kb, a => a.Content, new EnglishTokenizer());

// 2. Retrieve context for LLM
var query = userQuestion;
var context = index.Search(query, topK: 5)
    .Select(r => r.document.Content)
    .ToList();

// 3. Pass to LLM
var llmPrompt = $"Context:\n{string.Join("\n", context)}\n\nQuestion: {query}";
var response = await llm.GenerateAsync(llmPrompt);

Hybrid Search (Semantic + BM25)

// BM25 retrieval
var bm25Results = index.Search(query, topK: 20);

// Vector search (your embedding model)
var vectorResults = await vectorStore.SearchAsync(embedding, topK: 20);

// Hybrid ranking (combine scores)
var hybrid = bm25Results
    .Union(vectorResults)
    .GroupBy(r => r.id)
    .Select(g => new {
        doc = g.Key,
        score = g.Sum(x => x.score)  // Combine scores
    })
    .OrderByDescending(x => x.score)
    .Take(10);

Knowledge Base Search

public class KnowledgeBaseSearch
{
    private readonly Bm25Index<Article> _index;
    
    public KnowledgeBaseSearch(List<Article> articles)
    {
        _index = new Bm25Index<Article>(
            articles,
            a => $"{a.Title} {a.Content}",
            new EnglishTokenizer()
        );
    }
    
    public List<Article> Find(string query, int limit = 5)
    {
        return _index.Search(query, topK: limit)
            .Select(r => r.document)
            .ToList();
    }
}

🐛 Troubleshooting

Empty search results?

• Verify documents contain indexed content
• Check tokenizer is splitting terms correctly
• Use ExplainScore() to debug scoring

Slow search on large indexes?

• Increase topK threshold (retrieve more before filtering)
• Use SearchBatch() for multiple queries
• Consider optimizing with Bm25Tuner

Low relevance scores?

• Adjust BM25 parameters (use Bm25Parameters.Aggressive for large corpora)
• Try EnglishTokenizer instead of SimpleTokenizer
• Run parameter tuning on your validation set

Out of memory?

• Reduce document count or document length
• Use streaming indexing (add documents incrementally)
• Consider splitting index across multiple instances

📖 Documentation

• 📘 Getting Started — 5-minute walkthrough
• 🔍 Advanced Usage — Tuning, persistence, integration
• 🏗️ Architecture — Algorithm deep dive
• 📚 API Reference — Complete API documentation
• 🤝 Contributing — Build, test, contribute
• 📋 Changelog — Version history and release notes

🧪 Testing

cd tests/ElBruno.BM25.Tests
dotnet test

Test Coverage:

• 70+ unit tests covering all features
• Performance benchmarks
• Edge cases (empty queries, large indexes, etc.)
• Persistence and serialization

📝 License

MIT License. See LICENSE for details.

💡 Use Cases

• ✅ RAG Systems — Retrieval-augmented generation for LLMs
• ✅ Knowledge Bases — Internal documentation search
• ✅ Hybrid Search — Combine with semantic/vector search
• ✅ Full-Text Search — Replace expensive Lucene/Elasticsearch for small-medium indexes
• ✅ Chatbot Context — Fast retrieval for conversational AI
• ✅ Content Discovery — Lightweight search for dynamic content
• ✅ Information Retrieval — Academic projects, research

🙏 Credits

• BM25 Algorithm — Stephen Robertson, Karen Zaragoza
• Porter Stemmer — Martin Porter
• Implementation — Bruno Capuano

Made with ❤️ for .NET developers who need fast, lightweight full-text search.