rlm 1.0.6

RLM CLI (Recursive Language Models Context Tool)

A .NET CLI tool for processing large documents that exceed LLM context windows, based on the Recursive Language Models paper. RLM implements a recursive data ingestion pattern, enabling streaming content decomposition, multi-turn processing, and result aggregation.

Key Features

• Multi-format support - Markdown, PDF, HTML, JSON, Word (.docx), plain text
• 6 chunking strategies - Uniform, filtering, semantic, token-based, recursive, auto
• Stateful sessions - Persistent session state for multi-turn processing
• Claude Code support - RLM Agent and Skill

Projects in Solution

Quick Start

Install from NuGet

# Requires .NET 10+
dotnet tool install -g rlm

# Update to latest version
dotnet tool update -g rlm

Setup Claude Code Integration

To use RLM with Claude Code, copy the skills and agent files to your project:

# Clone the repository (or download the .claude folder)
git clone https://github.com/endjin/RLM.git

# Copy the .claude folder to your project
cp -r RLM/.claude /path/to/your/project/

This provides:

• Skill (.claude/skills/rlm/) - Invoke with /rlm to process large documents
• Agent (.claude/agents/rlm-worker/) - Parallel chunk processing worker for Claude Code

Build and Run

# Build
dotnet build Solutions/Rlm.slnx

# Run directly
dotnet run --project Solutions/Rlm.Cli -- load document.md

# Run tests
dotnet test --solution Solutions/Rlm.slnx

Install as Global Tool (from source)

cd Solutions/Rlm.Cli
dotnet pack
dotnet tool install --global --add-source ./bin/Release rlm

Basic Workflow

# Load a document
rlm load large-document.md

# Check document info
rlm info

# Chunk for processing
rlm chunk --strategy uniform --size 50000

# Process chunks iteratively
rlm next                          # Get next chunk
rlm store chunk_0 "result..."     # Store partial result
rlm next                          # Continue until done

# Aggregate results
rlm aggregate

Supported Formats

Commands Reference

Recursive RLM & Parallel Processing

RLM supports recursive decomposition through session isolation and raw output modes.

Session Management

Use the global --session <id> flag to isolate state for parallel or recursive processes:

# Parent process
rlm load large.txt --session parent

# Child process (simulated recursion)
rlm load chunk_1.txt --session child_1
rlm chunk ... --session child_1

Scripting & Piping

Use --raw for pipe-friendly output without formatting:

# Get raw content of next chunk
rlm next --raw --session child_1 | process_script.sh

# Pipe content into store
echo "result" | rlm store chunk_1 - --session parent

Bulk Import

Import results from child sessions or external files:

# Import results directly from session files
rlm import "rlm-session-child_*.json" --session parent

# Import results from text files (key = filename)
rlm import "summaries/*.txt" --session parent

Chunking Strategies

Architecture Overview

Solutions/Rlm.Cli/
├── Commands/          # 13 CLI commands (load, chunk, next, etc.)
├── Core/
│   ├── Documents/     # Multi-format readers (PDF, HTML, Word, etc.)
│   ├── Chunking/      # 6 chunking strategies
│   ├── Validation/    # Input validation framework
│   ├── Processing/    # Chunk post-processors
│   ├── Output/        # JSON output models
│   └── Session/       # Persistent state management
└── Infrastructure/    # Session store, DI, JSON context

Dependencies

CLI Framework

• Spectre.Console / Spectre.Console.Cli - Rich console output and command parsing
• Spectre.IO - Testable file system operations

Document Processing

• PdfPig - PDF text extraction
• ReverseMarkdown - HTML to Markdown conversion
• DocumentFormat.OpenXml - Word document processing

Tokenization

• Microsoft.ML.Tokenizers - Accurate GPT-4 tokenization (cl100k_base)

Infrastructure

• Microsoft.Extensions.DependencyInjection - DI container
• Polly - Retry resilience for file operations

Documentation

Detailed documentation is available in .claude/skills/rlm/:

License

Apache-2.0