HarvestNet.Core 1.1.0

HarvestNet

A self-healing web scraping and crawling framework for .NET.

Every time a site you scrape changes its markup, your selectors break and your pipeline goes quiet until someone notices and fixes it by hand. HarvestNet adds an optional layer that catches this automatically: when a CSS selector stops matching, it hands the page to a free-tier LLM, asks for a replacement selector, checks that the replacement actually works, and remembers the fix so it never has to ask again for that field on that site.

Python has had this kind of tooling for a while (Scrapy, Scrapling, Crawlee and others). .NET has not: outside of raw HTML parsers like AngleSharp and HtmlAgilityPack, there has been no actively maintained, AI-assisted scraping framework built for C#. HarvestNet is an attempt to close that gap.

Why this exists

Scraping code rots. A site redesigns its product cards, renames a CSS class, or wraps a price in a new <span>, and every scraper pointed at that page silently starts returning nulls. Most teams find out when a report looks wrong, not when it happens. HarvestNet does not eliminate this problem, but it gives you a way to recover from it automatically instead of dropping everything to go read the new page source.

The healing step only runs when extraction actually fails, so a healthy scrape that keeps matching all its selectors never calls an LLM at all. Combined with a free tier like Groq's, this means the AI-assisted part of HarvestNet effectively costs nothing for most projects.

Features

• Polite crawling by default: per-host rate limiting, robots.txt checks, automatic retries with exponential backoff (or the server's own Retry-After header) on 429 and 5xx responses, and configurable concurrency.
• Two ways to define what to extract: a typed C# class with [HarvestField] attributes for compile-time safety, or a JSON field map for cases where the schema is only known at run time (this is what the CLI uses).
• CSS selectors, XPath, or regular expressions for any field, mixed freely within the same item.
• Self-healing selectors, powered by your choice of Groq, Gemini, a local Ollama model, or any OpenAI-compatible endpoint, with automatic fallback across multiple providers if you chain them. Healed selectors are cached to disk per host and field, so the cost of a fix is paid once.
• Optional browser rendering through HarvestNet.Browser (Playwright), for pages that need JavaScript to produce their final HTML.
• Resumable crawls: enable a checkpoint file and an interrupted run picks up where it left off instead of starting over.
• Proxy rotation across a pool of proxies, with shared credentials for gateway style proxy providers.
• Sitemap.xml seeding, so a crawl can discover its URLs instead of listing them by hand.
• Live progress reporting through a simple IProgress<HarvestProgress> callback.
• Three output sinks out of the box: JSON Lines, CSV and SQLite, all safe under concurrent writes.
• A CLI with no code required: describe a scrape as a JSON recipe and run it with harvestnet run recipe.json, or try a single selector against a live page with harvestnet test-selector.
• A clean library API for anything more custom: build a HarvestSpider<T>, add seed URLs, wire up sinks and a link extractor, and call RunAsync().

Installation

As a library

dotnet add package HarvestNet.Core

To scrape JavaScript-rendered pages, also add the browser rendering package and install Playwright's browser binaries once per machine:

dotnet add package HarvestNet.Browser
dotnet tool install --global Microsoft.Playwright.CLI
playwright install chromium

As a command line tool

dotnet tool install --global HarvestNet.Cli

This installs the harvestnet command.

(Both packages are not yet published to NuGet as of this writing. See the "Publishing" section below for how to push them from this repository.)

Quick start: the library

using HarvestNet.Core;
using HarvestNet.Core.Export;
using HarvestNet.Core.Extraction;

public class Quote
{
    [HarvestField(".text")]
    public string? Text { get; set; }

    [HarvestField(".author")]
    public string? Author { get; set; }
}

var spider = new HarvestSpider<Quote>()
    .AddSeedUrl("https://quotes.toscrape.com/")
    .WithItemSelector(".quote")
    .WithSink(new CsvSink<Quote>("quotes.csv"));

var summary = await spider.RunAsync();
Console.WriteLine($"Extracted {summary.ItemsExtracted} quotes from {summary.PagesFetched} pages.");

Run the full version of this example, including pagination, from samples/QuickStart.

Quick start: the CLI

Generate a starter recipe:

harvestnet init my-scrape

Edit my-scrape.json:

{
  "seedUrls": ["https://quotes.toscrape.com/"],
  "itemSelector": ".quote",
  "fields": {
    "text": { "selector": ".text" },
    "author": { "selector": ".author" }
  },
  "linkSelector": "li.next > a",
  "maxDepth": 5,
  "maxPages": 50,
  "output": { "format": "json", "path": "quotes.jsonl" }
}

Run it:

harvestnet run my-scrape.json

A more complete example, including self-healing, lives in samples/recipes/quotes-example.json.

Turning on self-healing

Self-healing is optional and off by default. It needs one LLM provider, all of which have a free tier or run entirely locally.

In code:

using HarvestNet.Core.Healing;

var healer = new SelectorHealer(
    new GroqHealingProvider(Environment.GetEnvironmentVariable("GROQ_API_KEY")!),
    new HealingCache());

var spider = new HarvestSpider<Quote>()
    .AddSeedUrl("https://quotes.toscrape.com/")
    .WithItemSelector(".quote")
    .WithHealing(healer)
    .WithSink(new CsvSink<Quote>("quotes.csv"));

In a recipe:

{
  "healing": {
    "provider": "groq",
    "apiKey": "env:GROQ_API_KEY",
    "model": "llama-3.3-70b-versatile"
  }
}

apiKey accepts either a literal key or env:VARIABLE_NAME to read it from an environment variable at run time, so you never have to commit a key to a recipe file.

If you have a paid API key for a stronger model, you can use it too: GeminiHealingProvider and OpenAiCompatibleHealingProvider both work with any model name your account has access to. HarvestNet never requires a paid key; it just does not get in the way if you have one.

Chaining providers is also supported, so a fast API can fall back to a local model:

var healer = new SelectorHealer(
    new HealingProviderChain(
        new GroqHealingProvider(groqApiKey),
        new OllamaHealingProvider()),
    new HealingCache());

In a recipe, chain providers with a comma: "provider": "groq,ollama". Note that a recipe's apiKey and model are shared across every provider in the chain, so this only works cleanly when at most one provider in the chain actually needs a key (Ollama does not). For chains where each provider needs its own key or model, build the chain in code instead.

XPath and regular expressions

Every field can use a CSS selector (the default), an XPath expression, or a regular expression, chosen independently per field:

public class Product
{
    [HarvestField(".product-title")]
    public string? Title { get; set; }

    [HarvestField("//span[@class='price']", Kind = SelectorKind.XPath)]
    public string? Price { get; set; }

    [HarvestField(".sku-text", Kind = SelectorKind.RegexOnText, Attribute = @"SKU-\d+")]
    public string? Sku { get; set; }
}

In a recipe, set "xpath": true on a field to treat its selector as XPath:

"price": { "selector": "//span[@class='price']", "xpath": true }

Item containers (the repeating element passed to WithItemSelector or itemSelector) are CSS only for now; XPath containers are on the roadmap. When a field's XPath is evaluated inside an item (rather than against the whole page), write it relative to that item using a leading .//, for example .//span[@class='price'], the same way you would in any XPath tool.

Scraping JavaScript-rendered pages

Some sites only produce their real content after JavaScript runs. For those, render with a real browser instead of a plain HTTP request:

using HarvestNet.Browser;

await using var renderer = new PlaywrightPageRenderer();

var spider = new HarvestSpider<Quote>()
    .AddSeedUrl("https://example.com/")
    .WithItemSelector(".quote")
    .WithBrowserRendering(renderer)
    .WithSink(new CsvSink<Quote>("quotes.csv"));

await spider.RunAsync();

In a recipe, set "useBrowserRendering": true. This needs the Playwright browser binaries installed once per machine (see Installation above); without them, rendering will fail with a clear error from Playwright telling you to run playwright install.

Rendering a page with a real browser is much slower than a plain HTTP request, so use it only for the sites that actually need it.

Resumable crawls

For long crawls that might get interrupted, enable a checkpoint file:

var spider = new HarvestSpider<Quote>()
    .AddSeedUrl("https://example.com/")
    .WithItemSelector(".quote")
    .WithCheckpoint("crawl-state.json")
    .WithSink(new CsvSink<Quote>("quotes.csv"));

If the process is killed partway through, running the same code again picks up from the last completed depth level instead of starting over. The checkpoint file is deleted automatically once a crawl finishes cleanly. The CLI does this for every recipe automatically, writing to <output path>.checkpoint.json unless you set checkpointPath yourself; if you re-run harvestnet run on a recipe with a leftover checkpoint, it resumes and tells you so.

Proxies

Rotate through a pool of proxies, one per request:

var options = new CrawlOptions
{
    ProxyPool = new List<string> { "http://proxy-a:8080", "http://proxy-b:8080" },
    ProxyUsername = "myuser",
    ProxyPassword = "mypassword"
};

The same username and password are used for every proxy in the pool, which covers the common case of a single login for a rotating proxy gateway. In a recipe, set proxyPool, proxyUsername and proxyPassword.

Seeding from a sitemap

Instead of listing every seed URL by hand:

using var httpClient = new HttpClient();
var urls = await SitemapReader.ReadUrlsAsync(new Uri("https://example.com/sitemap.xml"), httpClient);

var spider = new HarvestSpider<Quote>().AddSeedUrls(urls);

SitemapReader also follows sitemap index files (a sitemap that points to other sitemaps), up to a few levels deep. In a recipe, set sitemapUrl and its URLs are added to seedUrls automatically before the crawl starts.

Progress reporting

var progress = new Progress<HarvestProgress>(p =>
    Console.WriteLine($"{p.PagesFetched} pages, {p.ItemsExtracted} items so far"));

var spider = new HarvestSpider<Quote>()
    .AddSeedUrl("https://example.com/")
    .WithProgress(progress);

The CLI shows this as a live, updating counter while a recipe runs.

Trying a selector before writing a recipe

harvestnet test-selector https://example.com/product/123 ".price"
harvestnet test-selector https://example.com/product/123 "//span[@class='price']" --xpath
harvestnet test-selector https://example.com/product/123 "a.details" --attribute href

This fetches the page once and shows what the selector matches, without needing a full recipe file.

Output formats

Set output.format in a recipe (or pick a sink directly in code) to json, csv, or sqlite. All three sinks buffer writes safely across the concurrent requests HarvestNet makes, so you do not need to add your own locking.

How selector healing actually works

1. HarvestNet tries your CSS selector as usual.
2. If it finds nothing, and healing is enabled, it checks a local JSON cache (harvestnet-healed-selectors.json by default) for a previously healed selector for that host and field.
3. If nothing is cached, it sends a truncated HTML snippet, the field name, your optional description, and the old selector to the configured provider, and asks for a JSON response with a new selector.
4. The proposed selector is tried against the same page. If it actually finds something, it is used for that extraction and written to the cache.
5. If it does not find anything, or the provider call fails for any reason, HarvestNet falls back to leaving the field empty rather than crashing the crawl.

Because the healed selector is cached per host and field, the LLM is only called the first time a break is detected on a given site, not on every subsequent page or run.

Comparison

AngleSharp and HtmlAgilityPack are excellent HTML parsers and HarvestNet actually uses AngleSharp under the hood. What they do not give you is the crawling, retry, rate limiting, and self-healing layer on top, which is what HarvestNet adds.

Roadmap

• XPath support for item containers, not just individual fields.
• A pluggable dedupe/storage backend for very large crawls (beyond the JSON checkpoint file, which is fine for most projects but not built for millions of URLs).
• Per-provider API keys when chaining healing providers through a recipe.
• More healing providers as free-tier APIs come and go.

Contributions toward any of these are very welcome; see CONTRIBUTING.md.

Contributing

Bug reports, feature ideas and pull requests are all welcome. See CONTRIBUTING.md for how the project is organized and how to get set up locally.

License

MIT. See LICENSE.