Spidey 6.0.4

dotnet add package Spidey --version 6.0.4
                    
NuGet\Install-Package Spidey -Version 6.0.4
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Spidey" Version="6.0.4" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Spidey" Version="6.0.4" />
                    
Directory.Packages.props
<PackageReference Include="Spidey" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Spidey --version 6.0.4
                    
#r "nuget: Spidey, 6.0.4"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Spidey@6.0.4
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Spidey&version=6.0.4
                    
Install as a Cake Addin
#tool nuget:?package=Spidey&version=6.0.4
                    
Install as a Cake Tool

<img src="https://jacraig.github.io/Spidey/images/icon.png" style="height:25px" alt="Spidey Icon" /> Spidey

.NET Publish NuGet

Spidey is a flexible and extensible .NET library for crawling web content. It is designed for .NET Core applications and provides a modular architecture, allowing you to customize or extend any part of the crawling pipeline.

Features

  • Simple API for crawling websites
  • Highly configurable via the Options class
  • Dependency injection support (IoC/DI)
  • Easily replaceable subsystems (engine, parser, scheduler, etc.)
  • Callback-based result handling
  • NuGet package available

Quick Start

Install the NuGet package:

dotnet add package Spidey

Setting up the Library

Register Spidey in your app's service collection using the RegisterSpidey extension method:

using Microsoft.Extensions.DependencyInjection;
using Spidey;

var services = new ServiceCollection();
services.RegisterSpidey();

// Optionally, register your Options configuration
services.AddSingleton(new Options
{
    ItemFound = result => Console.WriteLine($"Found: {result.Url}"),
    Allow = new List<string> { "http://mywebsite", "http://mywebsite2" },
    FollowOnly = new List<string> { /* regex patterns */ },
    Ignore = new List<string> { /* regex patterns */ },
    StartLocations = new List<string> { "http://mywebsite", "http://mywebsite2" },
    UrlReplacements = new Dictionary<string, string> { /* { "old", "new" } */ },
    // Other options as needed
});

var provider = services.BuildServiceProvider();
var crawler = provider.GetRequiredService<Crawler>();

Alternatively, you can instantiate Crawler and Options directly without DI:

var options = new Options
{
    ItemFound = result => Console.WriteLine($"Found: {result.Url}"),
    // ...other options
};
var crawler = new Crawler(options);

Options Configuration

The Options class configures the crawler's behavior. Key properties include:

  • ItemFound (Action<ResultFile>): Callback invoked when a new page is discovered.
  • Allow (List<string>): Regex patterns for URLs allowed to be crawled.
  • FollowOnly (List<string>): Regex patterns for pages whose links should be followed.
  • Ignore (List<string>): Regex patterns for URLs to ignore.
  • StartLocations (List<string>): Initial URLs to start crawling from.
  • UrlReplacements (Dictionary<string, string>): URL replacements during crawling.
  • NetworkCredentials (NetworkCredential): Optional credentials for authentication.
  • UseDefaultCredentials (bool): Use default system credentials.
  • Proxy (IWebProxy): Optional proxy settings.

Example callback method:

void OnItemFound(ResultFile result)
{
    Console.WriteLine($"Discovered: {result.Url} (Status: {result.StatusCode})");
    // Additional processing...
}

Basic Usage

Once configured, start the crawl process:

crawler.StartCrawl();

The library will handle link discovery, content downloading, and result parsing. Your callback will be invoked for each discovered item.

Customization

Spidey is built with extensibility in mind. The system is divided into the following subsystems, each replaceable via DI:

  1. Content Parser (IContentParser) – Parses downloaded data into ResultFile objects.
  2. Engine (IEngine) – Handles HTTP requests and content downloading.
  3. Link Discoverer (ILinkDiscoverer) – Extracts links from content.
  4. Processor (IProcessor) – Processes parsed content (default: invokes your callback).
  5. Scheduler (IScheduler) – Manages work distribution.
  6. Pipeline (IPipeline) – Orchestrates the crawling process.

To customize, implement the relevant interface from Spidey.Engines.Interfaces and register your implementation in the service provider. Note that if you call RegisterSpidey(), the registration is handled for you automatically. If you instantiate Crawler directly, you must compose the pipeline manually.

FAQ

Q: Can I run the crawler on multiple nodes?

A: The default scheduler is single-node only. For distributed crawling, implement a custom scheduler (e.g., using a database or message queue) to coordinate work between instances.

Build Process

Requirements:

  • Visual Studio 2022

Clone the project and open the solution (Spidey.sln) in Visual Studio to build.

License

See LICENSE for details.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
6.0.4 131 7/16/2025
6.0.3 139 6/27/2025
6.0.2 146 6/25/2025
6.0.1 166 12/9/2024
6.0.0 148 11/25/2024
5.0.131 145 11/12/2024
5.0.130 128 11/11/2024
5.0.129 127 11/6/2024
5.0.128 123 11/5/2024
5.0.127 124 11/4/2024
5.0.126 126 10/31/2024
5.0.125 119 10/30/2024
5.0.124 133 10/29/2024
5.0.123 141 10/11/2024
5.0.122 133 10/10/2024
5.0.121 123 10/9/2024
5.0.120 134 10/2/2024
5.0.119 131 10/1/2024
5.0.118 145 9/24/2024
5.0.117 144 9/17/2024
5.0.116 180 9/10/2024
5.0.115 142 9/3/2024
5.0.114 134 8/30/2024
5.0.113 147 8/27/2024
5.0.112 152 8/26/2024
5.0.111 162 8/23/2024
5.0.110 161 8/21/2024
5.0.109 153 8/20/2024
5.0.108 161 8/16/2024
5.0.107 160 8/15/2024
5.0.106 138 8/5/2024
5.0.105 118 8/2/2024
5.0.104 130 8/1/2024
5.0.103 131 7/26/2024
5.0.102 153 7/11/2024
5.0.101 149 7/2/2024
5.0.100 149 6/27/2024
5.0.99 137 6/26/2024
5.0.98 158 6/19/2024
5.0.97 150 6/18/2024
5.0.96 157 6/17/2024
5.0.95 150 6/14/2024
5.0.94 139 6/13/2024
5.0.93 139 6/12/2024
5.0.92 146 5/31/2024
5.0.91 141 5/30/2024
5.0.90 148 5/17/2024
5.0.89 155 5/16/2024
5.0.88 167 5/8/2024
5.0.87 166 5/7/2024
5.0.86 181 5/6/2024
5.0.85 121 5/3/2024
5.0.84 140 5/2/2024
5.0.83 148 5/1/2024
5.0.82 162 4/30/2024
5.0.81 152 4/16/2024
5.0.80 154 4/12/2024
5.0.79 153 4/11/2024
5.0.78 168 4/1/2024
5.0.77 158 3/29/2024
5.0.76 163 3/18/2024
5.0.75 148 3/15/2024
5.0.74 157 3/14/2024
5.0.73 152 3/11/2024
5.0.72 147 3/8/2024
5.0.71 152 3/7/2024
5.0.70 173 3/6/2024
5.0.69 162 3/5/2024
5.0.68 158 3/4/2024
5.0.67 165 2/29/2024
5.0.66 151 2/28/2024
5.0.65 161 2/26/2024
5.0.64 158 2/23/2024
5.0.63 167 2/22/2024
5.0.62 165 2/21/2024
5.0.61 158 2/16/2024
5.0.60 157 2/15/2024
5.0.59 164 2/12/2024
5.0.58 155 2/8/2024
5.0.57 149 2/7/2024
5.0.56 152 2/6/2024
5.0.55 139 2/1/2024
5.0.54 158 1/31/2024
5.0.53 147 1/30/2024
5.0.52 141 1/24/2024
5.0.51 155 1/23/2024
5.0.50 159 1/12/2024
5.0.49 157 1/11/2024
5.0.48 166 12/26/2023
5.0.47 152 12/22/2023
5.0.46 147 12/18/2023
5.0.45 135 12/15/2023
5.0.44 136 12/14/2023
5.0.43 147 12/13/2023
5.0.42 143 12/12/2023
5.0.41 171 11/24/2023
5.0.40 163 11/21/2023
5.0.39 138 11/20/2023
5.0.38 140 11/17/2023
5.0.37 144 11/16/2023
5.0.36 142 11/14/2023
5.0.35 138 11/8/2023
5.0.34 130 11/7/2023
5.0.33 145 11/6/2023
5.0.32 142 11/1/2023
5.0.31 137 10/31/2023
5.0.30 160 10/30/2023
5.0.29 146 10/26/2023
5.0.28 180 10/12/2023
5.0.27 179 10/5/2023
5.0.26 165 9/26/2023
5.0.25 142 9/20/2023
5.0.24 140 9/19/2023
5.0.23 182 9/18/2023
5.0.22 158 9/14/2023
5.0.21 143 9/13/2023
5.0.20 170 9/11/2023
5.0.19 167 9/7/2023
5.0.18 164 9/6/2023
5.0.17 161 9/5/2023
5.0.16 175 9/4/2023
5.0.15 163 9/1/2023
5.0.14 172 8/31/2023
5.0.13 184 8/30/2023
5.0.12 186 8/29/2023
5.0.11 172 8/25/2023
5.0.10 187 8/23/2023
5.0.9 172 8/18/2023
5.0.8 195 8/10/2023
5.0.7 196 8/8/2023
5.0.6 201 8/8/2023
5.0.5 221 8/7/2023
5.0.4 200 8/3/2023
5.0.3 216 7/26/2023
5.0.2 192 7/20/2023
5.0.1 204 7/14/2023
5.0.0 365 12/12/2022
4.0.5 605 6/10/2022
4.0.2 566 1/11/2022
4.0.1 535 7/19/2021
3.0.9 596 1/7/2021
3.0.7 681 9/13/2020
3.0.6 654 6/26/2020
3.0.5 631 6/26/2020
3.0.3 639 3/25/2020
3.0.2 696 3/1/2020
3.0.1 714 1/1/2020
3.0.0 685 12/23/2019
2.0.15 721 11/22/2019
2.0.14 676 11/22/2019
2.0.13 666 11/22/2019
2.0.12 655 11/21/2019
2.0.11 651 11/21/2019
2.0.10 663 11/21/2019
2.0.9 653 11/21/2019
2.0.8 834 3/3/2019
2.0.7 760 3/3/2019
2.0.6 777 3/3/2019
2.0.5 761 3/3/2019
2.0.4 822 2/7/2019
2.0.3 1,306 6/1/2018
2.0.2 1,295 5/22/2018
2.0.1 1,431 1/2/2018
1.0.12 1,203 11/2/2017
1.0.11 1,200 10/30/2017
1.0.10 1,205 10/26/2017
1.0.9 1,201 10/26/2017
1.0.8 1,233 10/26/2017
1.0.7 1,187 10/25/2017
1.0.6 1,175 10/25/2017
1.0.5 1,215 10/24/2017
1.0.4 1,151 10/24/2017
1.0.3 1,147 10/19/2017
1.0.2 1,280 10/18/2017
1.0.1 1,201 9/29/2017