EvaluatedApplications.Prose 1.3.3

Prose

Beta. The public API may change between versions while we improve it. Breaking changes always come with a new major version, and the release notes say what changed.

A grammar-driven synthetic corpus generator for .NET. Prose reads plain text through the rules of grammar (a deterministic, zero-training parser), keeps what it learns in a HoloDb database, and recombines it into new sentences that are grammatical, use words suited to their slots, and read plausibly — plus reading-comprehension Q&A pairs. It exists to grow a training corpus: mine the knowledge already in your text, then generate more of it.

It works the way a person who has learned grammar does — find the subject, the verb, which adjectives modify which noun — rather than by pasting fragments together. No word is locked to one role: shoots is a noun after "bamboo" and a verb after "panda", decided by context. The canonical example is Eats, Shoots & Leaves: the comma changes the parse, and the meaning with it (see Panda contrast below).

This document is the only documentation shipped with the NuGet package (binary-only, no source, no docs site) — every public type and method is covered below with a minimal, real, compiled example. Nothing in this package is gated behind a tier or a license key.

dotnet add package EvaluatedApplications.Prose

Table of contents

• Quick start
• Panda contrast
• Pipeline overview
• ProseEngine — the facade
• Grammar types (Pos, Role, Tok, Chunk, ChunkKind, Clause, ParsedSentence)
• Low-level pipeline classes (WordTokenizer, SentenceSplitter, TextSource, PosTagger, Chunker, Roles, Lexicons, Morph)
• Mining & knowledge (ProseStore, WordFlag, Knowledge)
• Generation (Generator, GenClause, GenKind, SubjectSpec, DetKind, Realizer, ValidityGate, Attested)
• Question/answer generation (QaPair, QaMaker, Passage, PassageMaker)
• Plausibility scorer (IPlausibility, PlausFormer)
• Gotchas
• License

Quick start

using Prose;

var prose = new ProseEngine();
prose.MineText("The wise king rules the peaceful kingdom. The kingdom is calm. " +
                "The king gives the poor man a book. The man reads the book. " +
                "The dog chases the red ball. The dog barks. The sky is blue. " +
                "The queen sends the man a letter. The queen is kind.");

foreach (var sentence in prose.Generate(5, seed: 1))
    Console.WriteLine(sentence);

Compiled and run against the current build; actual output for this exact text and seed (small corpora recombine freely and can read oddly — see Gotchas):

A king rules the peaceful kingdom.
The peaceful ball barks.
A queen sends the letter.
A book chases the ball.
The peaceful kingdom rules the wise kingdom.

For real use, mine a folder of many real .txt files (prose.Mine(@"C:\corpus\text")) instead of a single hand-written string — a larger, more coherent corpus yields far more sensible output because every open-class word is drawn from, and every selectional pairing is filtered against, what was actually observed (see Knowledge.Strict).

Panda contrast

The acceptance target for the whole rules engine — the same words, two different punctuation marks, two different parses:

foreach (var s in ProseEngine.Parse("The panda eats, shoots, and leaves. The panda eats shoots and leaves."))
    Console.WriteLine(ProseEngine.Explain(s));

Output:

The/Det panda/Noun eats/Verb shoots/Verb and/Cc leaves/Verb
template: DET NOUN VERB_FIN , VERB_FIN , CC VERB_FIN
  subject=panda  verbs=[eats,shoots,leaves]  indirectObject=-  objects=[-]  predicate=-
The/Det panda/Noun eats/Verb shoots/Noun and/Cc leaves/Noun
template: DET NOUN VERB_FIN NOUN CC NOUN
  subject=panda  verbs=[eats]  indirectObject=-  objects=[shoots,leaves]  predicate=-

With the comma, "shoots" and "leaves" are coordinate finite verbs (the panda eats, [and] shoots, [and] leaves — three actions). Without it, "eats" is the only verb and "shoots"/"leaves" are its two coordinate direct objects (the panda eats [shoots and leaves], a diet). No word is ever locked to one part of speech; the frame around it decides.

Pipeline overview

1. Parse (rules, zero training). Sentence splitting, a closed-class + morphology + distributional POS tagger, phrase chunking, and role assignment (subject, coordinate finite verbs, objects, indirect object, copular predicate).
2. Mine → HoloDb. A lexicon, subject→verb / verb→object / verb→indirect-object / adjective→noun selectional tables, syntax templates and bigrams, written with BulkLoad.
3. Generate. Sample a clause shape from the mined templates, fill each slot with a word that is the right category, suitable (respects the selectional tables) and plausible, realize it with correct agreement (a/an, subject–verb), and emit only sentences that pass a validity gate (which re-parses each candidate).
4. Plausibility (optional). A small HoloFormer language model over content-word lemmas ranks candidates so meaning improves while grammar stays guaranteed.
5. Q&A pairs. Because each generated sentence keeps its role structure, Prose emits reading-comprehension pairs — a context, a question, and the answer derivable from it — including multi-sentence passages with coreference and self-referential two-turn exchanges.

ProseEngine — the facade

The type almost every consumer uses. One instance owns one in-memory ProseStore; mine into it, then generate from it.

Mining from a folder

var dir = @"C:\corpus\text";               // folder of *.txt files
var prose = new ProseEngine();
prose.Mine(dir);                            // default pattern "*.txt"
Console.WriteLine(prose.Store.SentencesMined);

Persisting to HoloDb

var db = prose.Save(@"C:\data\prose.wal");  // writes prose.wal (+ prose.wal.snap) to disk
// db is a HoloDb.Database — query the mined tables (lexicon, sv, vo, io, an, templates, bigrams)
// with HoloDb's own SQL surface: db.Execute("SELECT * FROM lexicon LIMIT 10")

var memDb = prose.Save();                   // null path = in-memory, nothing touches disk

Training the plausibility scorer and generating with it

double loss = prose.TrainPlausibilityText(corpusText, epochs: 3);
Console.WriteLine($"loss={loss:F4} trained={prose.Plausibility?.Trained}");
foreach (var s in prose.Generate(5, seed: 2))
    Console.WriteLine(s);

Real output against the quick-start corpus:

loss=3.1671 trained=True
The queen sends.
A book chases a red ball.
A dog chases a red ball.
A queen sends the man the letter.
A queen sends a letter.

Writing a corpus to disk

int written = prose.WriteCorpus(@"C:\corpus\synthetic", 100_000);

Q&A generation

foreach (var pair in prose.GenerateQa(3, seed: 4))
    Console.WriteLine($"{pair.Context} | {pair.Question} -> {pair.Answer}");

Real output (against the quick-start corpus):

A letter is wise. | What is a letter? -> Wise.
A letter is wise. | What is wise? -> A letter.
The poor man reads. | What reads? -> The poor man.

foreach (var pair in prose.GenerateQaPassages(2, sentencesPerPassage: 3, seed: 5))
    Console.WriteLine($"{pair.Question} -> {pair.Answer}");
// What reads the book? -> The man.
// What kind of man reads the book? -> Poor.

foreach (var (prompt, target) in prose.GenerateTwoTurnQa(2, seed: 6))
    Console.WriteLine($"{prompt} => {target}");

int n = prose.WriteQaPairs(@"C:\corpus\qa", 50_000);   // writes prose-qa.tsv: prompt\ttarget

Parsing without mining or a database

var p = ProseEngine.ParseSentence("The wise king rules the kingdom.");
Console.WriteLine(ProseEngine.Explain(p));

The/Det wise/Adj king/Noun rules/Verb the/Det kingdom/Noun
template: DET ADJ NOUN VERB_FIN DET NOUN
  subject=king  verbs=[rules]  indirectObject=-  objects=[kingdom]  predicate=-

Grammar types

Plain data types returned by parsing. You read these; you rarely construct them yourself (Tok has a public constructor, Chunk/Clause/ParsedSentence are populated by the pipeline).

var p = ProseEngine.ParseSentence("The dog chases the ball.");
foreach (var t in p.Words) Console.WriteLine($"{t.Text} -> {t.Pos} (role {t.Role}, lemma {t.Lemma})");
Console.WriteLine(p.Template);          // "DET NOUN VERB_FIN DET NOUN"
Console.WriteLine(p.Clauses[0].Subject);   // token index of "dog"

Low-level pipeline classes

ProseEngine.Parse/ParseSentence already run every one of these stages for you — reach for them directly only if you need a single stage in isolation (e.g. tokenizing without tagging, or tagging without chunking).

WordTokenizer

Splits a sentence string into word/punctuation Toks; splits off clitics (n't, 's, 're...).

var toks = WordTokenizer.Tokenize("The dog chases the red ball.", sentenceIndex: 0);

SentenceSplitter

Splits raw text into sentences, guarding abbreviations ("Mr. Smith") and decimals ("3.14").

foreach (var s in SentenceSplitter.Split("Mr. Smith ran. He stopped at 3.14 meters."))
    Console.WriteLine(s);
// Mr. Smith ran.
// He stopped at 3.14 meters.

TextSource

Reads a corpus folder: Files(dir, pattern), ReadClean(path), Clean(text) (strips byte-escape junk and Gutenberg _italics_ underscores), Sentences(dir, pattern) (streamed, file by file — what ProseEngine.Mine calls internally).

Console.WriteLine(TextSource.Clean("He said \\x93hi\\x94 to _them_."));
// He said  hi  to  them .
foreach (var s in TextSource.Sentences(@"C:\corpus\text")) { /* ... */ }

PosTagger

static void Tag(List<Tok> toks) — the four-pass tagger (base candidates, closed-class disambiguation, open-class resolution with clause state, coordination). Mutates toks in place.

var toks = WordTokenizer.Tokenize("The dog chases the red ball.", 0);
PosTagger.Tag(toks);

Chunker

static List<Chunk> Chunk(List<Tok> toks) — bottom-up NP/VP/PP/RelCl chunking over already-tagged tokens.

var chunks = Chunker.Chunk(toks);   // toks must already be tagged (PosTagger.Tag)

Roles

static (List<Clause> clauses, string template) Extract(List<Tok> toks, List<Chunk> chunks) — subject/verb/object/predicate assignment over the chunks; also tags each Tok.Role as a side effect. static string Sym(Pos p, bool finiteVerb = false) — the template symbol for a tag ("VERB_FIN" vs "VERB", "NOUN", "DET", ...).

var (clauses, template) = Roles.Extract(toks, chunks);
Console.WriteLine(template);                         // "DET NOUN VERB_FIN DET ADJ NOUN"
Console.WriteLine(Roles.Sym(Pos.Verb, finiteVerb: true));   // "VERB_FIN"

Lexicons

The closed-class word lists and small lexical gates. Public read-only HashSet<string> fields — Det, Qnt, PronSubj, PronObj, PronPoss, PronRefl, PronIndef, Prep, Cc, Sub, AuxBe, AuxHave, AuxDo, Modal, Neg, Wh, Particle, ArchaicAdv, CommonAdv, MassNoun — plus:

Console.WriteLine(Lexicons.IsDet("the"));        // True
Console.WriteLine(Lexicons.IsCopula("became"));  // True
Console.WriteLine(Lexicons.IsWordLike("cafe"));  // True
Console.WriteLine(Lexicons.IsWordLike("fld"));   // False (no vowel)
Console.WriteLine(string.Join(",", Lexicons.Closed("that")));   // Det,Sub

Morph

Morphology: suffix-based open-class voting, lemma recovery, and the realizer's inflections. Public Dictionary<string,string> IrregPast (surface→lemma for ~90 irregular simple-past forms, e.g. IrregPast["gave"] == "give"), plus:

Console.WriteLine(Morph.Lemma("running", Pos.Verb));   // run
Console.WriteLine(Morph.Lemma("buses", Pos.Noun));     // bus
Console.WriteLine(Morph.ThirdPerson("eat"));           // eats
Console.WriteLine(Morph.Pluralize("box"));             // boxes
Console.WriteLine(Morph.Article("apple"));             // an
Console.WriteLine(Morph.LooksPlural("buses"));         // True
Console.WriteLine(Morph.LooksPlural("bus"));           // False — curated exception, NOT "bu" + plural
Console.WriteLine(Morph.IrregPast["gave"]);            // give

Mining & knowledge

ProseStore

Accumulates mined counts in memory while scanning, then writes each HoloDb table exactly once (BulkLoad; HoloDb has no UPSERT, so counts are never written per-row).

var store = new ProseStore();
store.Add(ProseEngine.ParseSentence("The dog chases the ball."));
store.Add(ProseEngine.ParseSentence("The cat chases the mouse."));
Console.WriteLine(store.SentencesMined);   // 2
var knowledge = store.BuildKnowledge();

WordFlag

[Flags] public enum WordFlag { None = 0, Transitive = 1, Mass = 2, Gradable = 4, Proper = 8, Countable = 16 }

Per-word metadata bits stored in the mined lexicon HoloDb table's flags column (readable via db.Execute("SELECT * FROM lexicon") after Save). Combine with the usual flag operators:

var flags = WordFlag.Transitive | WordFlag.Countable;

Knowledge

The sampling-ready, in-RAM view ProseStore.BuildKnowledge() returns; the Generator reads exclusively from this. Public fields: ByPos (word/lemma/freq per Pos), VerbsForSubject, ObjsForVerb, IndirectObjsForVerb (dative), AdjsForNoun, NounForms/VerbForms (observed surfaces per lemma), Templates (syntax-template frequency), AdjSurfaces, TransitiveVerbs, DitransitiveVerbs, Bigrams, Vocab (every observed surface — the attestation gate), Strict (true once ≥500 sentences were mined — generation then draws ONLY from observed pairings, no random fallback). Plus void Add(map, k, v, cnt) and List<(...)> Pos(Pos p) helpers.

var k = store.BuildKnowledge();
Console.WriteLine($"Vocab={k.Vocab.Count} Strict={k.Strict}");

Generation

Generator

Recombines mined Knowledge into new sentences: picks a clause shape weighted by the corpus's own templates, fills each slot with a suitable, plausible, attested word, realizes agreement, and only emits sentences that pass ValidityGate.

var gen = new Generator(k, seed: 7);
var clause = gen.BuildClause();
Console.WriteLine(clause?.Text);     // "The queen sends the letter."
Console.WriteLine(gen.Next());       // "The book is calm."
foreach (var s in gen.Generate(3)) Console.WriteLine(s);

var forced = new SubjectSpec("king", "king", Plural: false, Det: DetKind.The);
var c2 = gen.BuildClause(forced);
Console.WriteLine(c2?.Text);         // "The king rules the kingdom."

GenKind, SubjectSpec, DetKind

public enum GenKind { Intransitive, Transitive, Copular, Ditransitive }
public readonly record struct SubjectSpec(
    string Surface, string Lemma, bool Plural, DetKind Det, bool Pronoun = false, string? Antecedent = null);
public enum DetKind { The, Indef, Some, None }

GenKind is GenClause.Kind's value, telling you which clause shape was built. SubjectSpec is the forced-subject argument to Generator.BuildClause (used internally by PassageMaker to make later sentences reuse an earlier entity, either as a definite NP or a pronoun). DetKind selects which determiner Realizer.Np emits.

GenClause

The sentence plus its full role structure, returned by Generator.BuildClause and consumed by QaMaker. Public fields: Text, Lemmas, SubjectNp, SubjectPlural, SubjectSurface, SubjectLemma, SubjectAntecedent, VerbLemma, VerbSurface, VerbIsPast, Kind, ObjectNp/ObjectSurface/ObjectLemma/ObjectPlural, IndirectObjectNp/IndirectObjectSurface/IndirectObjectLemma/IndirectObjectPlural, Predicate, SubjectAdj/ObjectAdj/IndirectObjectAdj (the attested adjective attached to that NP, if any).

Realizer

Turns chosen lemmas/surfaces into text, honouring agreement/article rules. Never synthesizes a noun or verb inflection — only the determiner (a function word) is chosen here.

Console.WriteLine(Realizer.Np("apple", "red", DetKind.Indef));   // "a red apple"
Console.WriteLine(Realizer.Capitalize("hello world"));           // "Hello world"

ValidityGate

public static bool Check(string sentence, out string reason);
public static bool Check(string sentence);   // reason discarded

The §8 validity gate: re-parses a candidate sentence with the SAME rules that mined the corpus and only accepts it if that re-parse finds a real subject + finite verb, no double determiner, no "a cats"-style indefinite+plural, and subject–verb number agreement (A1).

bool ok = ValidityGate.Check("The dog chases the ball.", out var reason);     // true, reason=""
bool bad = ValidityGate.Check("A cats runs.");                                 // false

Attested

public static bool All(string text, Knowledge k);

The "nothing imaginary" backstop: every open-class word in text must be a surface the corpus actually used (Knowledge.Vocab); closed-class function words are always allowed. Generator and QaMaker's callers already run every candidate through this — call it yourself only if you build text outside the normal Generate/GenerateQa* path.

bool allGood = Attested.All("The dog chases the ball.", k);

Question/answer generation

QaPair

public sealed record QaPair(string Context, string Question, string Answer);

A reading-comprehension triple: Answer is always derivable from Context.

QaMaker

var c = gen.BuildClause();
if (c != null)
    foreach (var qa in QaMaker.From(c, k))
        Console.WriteLine($"{qa.Question} -> {qa.Answer}");
// What chases a red ball? -> The dog.

Passage / PassageMaker

public sealed class Passage { public List<GenClause> Clauses; public string Text; }
public sealed class PassageMaker
{
    public PassageMaker(Knowledge k, int seed = 1);
    public Passage? Build(int sentences = 4);
}

Builds a coherent multi-sentence passage: the opening sentence introduces entities, and later sentences reuse them (a definite reference or a tracked-antecedent pronoun).

var pm = new PassageMaker(k, seed: 3);
var passage = pm.Build(3);
if (passage != null)
{
    Console.WriteLine(passage.Text);
    // A ball rules a kingdom. A blue king gives a man the book. It reads the book.
    foreach (var qa in QaMaker.FromPassage(passage, k))
        Console.WriteLine($"{qa.Question} -> {qa.Answer}");
    // What rules a kingdom? -> The ball.
    // What reads the book? -> The book.
}

Plausibility scorer

IPlausibility

public interface IPlausibility { double Score(IReadOnlyList<string> contentLemmas); }

Scores how plausible a content-word (lemma) sequence is — higher is more plausible. Implement this yourself to plug a different ranking model into Generator's scorer parameter.

PlausFormer

The shipped implementation: a small mini-HoloFormer language model over content-word lemmas. Rules guarantee grammaticality; this only ranks meaning. A POS tag is never decided by this model — tagging stays 100% rules-based.

var store = new ProseStore();
store.Mine(SentenceSplitter.Split(corpusText));
var plaus = PlausFormer.ForStore(store);

var seqs = new List<IReadOnlyList<string>>
{
    new List<string> { "dog", "chase", "ball" },
    new List<string> { "king", "rule", "kingdom" },
};
double trainLoss = plaus.Train(seqs, epochs: 3);
double score = plaus.Score(new List<string> { "dog", "chase", "ball" });
Console.WriteLine($"Vocab={plaus.Vocab} Trained={plaus.Trained} TrainLoss={trainLoss:F4} Score={score:F4}");
// Vocab=24 Trained=True TrainLoss=2.1179 Score=-0.5688

ProseEngine.TrainPlausibility/TrainPlausibilityText build and train one of these for you and wire it into Generate/GenerateQa* automatically — call the constructor directly only if you need a scorer decoupled from a ProseEngine instance.

Gotchas

• Determinism. Every generation/Q&A method takes a seed; the same Knowledge + the same seed always produces the same output sequence (the generator and the plausibility HoloFormer both use System.Random(seed)). Changing the corpus, the mining order, or the package version can still change output even with the seed held fixed.
• Attested-forms-only. Every open-class word in emitted output is an observed corpus surface (Knowledge.Vocab); nothing is ever synthesized. On a tiny corpus (fewer than ~500 mined sentences, Knowledge.Strict == false) the generator falls back to the full POS pool when no selectional pairing was observed — this is what produces odd-but-grammatical output like "The red ball chases." or "A book sends the letter." in the quick-start example above. On a real corpus (Strict == true), generation draws ONLY from observed subject→verb / verb→object / adjective→noun pairings, so this effect disappears. Plausibility training does not change which words CAN appear — it only re-ranks candidates that already passed every grammar/attestation gate (best-of-bestOf); it cannot fix a nonsensical pairing the small corpus never filtered out in the first place.
• HoloDb file locations. ProseEngine.Save(walPath)/ProseStore.Save(walPath) create (or open) walPath plus a walPath + ".snap" snapshot file; Save(null) is purely in-memory and touches no disk. Mining itself never touches HoloDb — only Save does. The returned HoloDb.Database is a live handle; query it with HoloDb's own SQL (db.Execute("SELECT ... FROM lexicon")) — see EvaluatedApplications.HoloDb's own package docs for the full SQL surface. HoloDb has no UPSERT, so re-mining into the same store and calling Save again on the same path appends a fresh BulkLoad, not a merge.
• Performance. Mining and generation are single-threaded, CPU-only, and allocate per sentence/candidate — fine interactively, but a full-corpus mine (millions of sentences) or a large WriteCorpus/WriteQaPairs run is a batch job to run in your own process/console, not something to await inline in a request handler. Generate/GenerateQa* internally retry rejected candidates up to a fixed multiple of n (e.g. n * 40 + 50 for Generate) before giving up, so a request for more sentences than the corpus can plausibly support returns fewer than n, not an infinite loop or an exception.
• Thread-safety. ProseEngine, ProseStore, and Generator are not thread-safe for concurrent writers — mine into one instance from one thread at a time. A Knowledge returned by BuildKnowledge() is an immutable snapshot safe to read from multiple Generator/PassageMaker instances concurrently (each owns its own System.Random).
• Exceptions. The public surface mostly returns null/empty rather than throwing on "couldn't build": Generator.Next()/BuildClause() return null when no valid candidate could be formed; QaMaker.TwoTurn returns null when the two clauses don't form a clean two-turn; ValidityGate.Check returns false (with a reason string) rather than throwing. Genuine exceptions come from the .NET BCL underneath (e.g. Mine on a missing folder returns no sentences rather than throwing — TextSource.Files yields nothing if the directory doesn't exist; File.WriteAllText in WriteCorpus/WriteQaPairs throws the normal System.IO exceptions for an unwritable path) or from HoloDb itself (duplicate primary key, bad SQL) on Save.
• Nothing is license-gated. Every method, type, and option documented above is available in every build of this package; there is no "Pro" tier.
• Obsolete APIs. None — every public member listed above is current as of this version.

License

Proprietary — this package ships the compiled library only; see LICENSE for the full terms. Installing and using unmodified copies for internal evaluation and non-commercial development or testing is licensed; commercial, production, or revenue-generating use requires a separate commercial licence from the owner. No feature on this page is gated behind a tier or a licence key — every capability described works identically with or without one. Copyright (c) 2026 Dongyang Stephen Chen, trading as Evaluated Applications. All rights reserved.