Spidey 6.0.1

dotnet add package Spidey --version 6.0.1
                    
NuGet\Install-Package Spidey -Version 6.0.1
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Spidey" Version="6.0.1" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Spidey" Version="6.0.1" />
                    
Directory.Packages.props
<PackageReference Include="Spidey" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Spidey --version 6.0.1
                    
#r "nuget: Spidey, 6.0.1"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=Spidey&version=6.0.1
                    
Install Spidey as a Cake Addin
#tool nuget:?package=Spidey&version=6.0.1
                    
Install Spidey as a Cake Tool

Spidey

.NET Publish

Library to help with crawling web content. Compatible with .Net Core and .Net Framework.

Setting up the Library

Spidey needs to be added to your app's service collection in order to be wired up properly. This is done by calling the RegisterSpidey extension method on the ServiceCollection object. This is done like so:

new ServiceCollection().RegisterSpidey();

Or if you're using Canister:

new ServiceCollection().AddCanisterModules();

The RegisterSpidey function is an extension method that registers it with the IoC container. When this is done, Spidey is ready to use.

Basic Usage

Spidey really boils down to using one class called Crawler:

ServiceCollection.AddSingleton(new Options
			{
			    ItemFound = FoundFile=>{}                                            //The callback method used when a new page is discovered.
			    Allow = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Regexes of what sites/pages are allowed to be crawled.
			    FollowOnly = new List<string> { "..." },                                 //Regexes of pages to only follow links that are found on them.
			    Ignore = new List<string> { "..." },                                     //Regexes that the system will ignore when they are encountered.
			    StartLocations = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Starting URLs for the crawler.
			    UrlReplacements = new Dictionary<string,string> {...}                    //When the system hits one of the keys in the dictionary, it will replace it with the value.
			});
			

Note that it is recommended that you actually register the Options object in your ServiceCollection and resolve the Crawler object from the service provider but it is not required. You can simply new up an instance of Crawler if you want. Anyway, the Options class has a number of properties, some of which are not displayed above such as NetworkCredentials, UseDefaultCredentials, and Proxy. The callback method is what will be called by the system once a link's info has been received and looks like this:

void CallbackMethod(ResultFile obj) { ... }

The library will handle parsing of links found within the page, downloading the content, etc. for the most part. At this point all you have to do is call the StartCrawl method:

MyCrawler.StartCrawl();

Customization

Note that it's possible to customize the crawler's various parts. The system is divided into the following sections:

  1. Content Parser (IContentParser) - This parses the resulting data and converts it to the ResultFile object.
  2. Engine (IEngine) - This downloads the content from the server.
  3. Link Discoverer (ILinkDiscoverer) - Takes the content from the engine and looks for links to other resources.
  4. Processor (IProcessor) - Takes the parsed content and hands it off to your code. The default one simply calls the method provided in the options.
  5. Scheduler (IScheduler) - Handles handing out work to the various workers.
  6. Pipeline (IPipeline) - Manages the various parts of the process by feeding the content to the next bit of the process.

These subsystems all implement interfaces found in the Spidey.Engines.Interfaces namespace. In order to replace the default in any of these systems all you need to do is create a class that implements the interface that you want to replace. After that the system will automatically pick it up if resolved from the service provider. If you, instead, new up a Crawler object then you will need to compose the Pipeline object.

Installation

The library is available via Nuget with the package name "Spidey". To install it run the following command in the Package Manager Console:

Install-Package Spidey

FAQ

  1. Is it possible to run the crawler using multiple nodes?

The default scheduler assumes that you are only running the crawler from one location and doesn't talk to other instances of the application. But it is possible to replace the scheduler with one that will talk via some mechanism like a database to coordinate work between instances and is recommended for more complex setups.

Build Process

In order to build the library you will require the following:

  1. Visual Studio 2022

Other than that, just clone the project and you should be able to load the solution and build without too much effort.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
6.0.1 116 4 months ago
6.0.0 117 4 months ago
5.0.131 106 5 months ago
5.0.130 94 5 months ago
5.0.129 97 5 months ago
5.0.128 96 5 months ago
5.0.127 88 5 months ago
5.0.126 96 5 months ago
5.0.125 85 5 months ago
5.0.124 95 5 months ago
5.0.123 102 6 months ago
5.0.122 96 6 months ago
5.0.121 88 6 months ago
5.0.120 97 6 months ago
5.0.119 100 6 months ago
5.0.118 103 6 months ago
5.0.117 111 7 months ago
5.0.116 148 7 months ago
5.0.115 108 7 months ago
5.0.114 106 7 months ago
5.0.113 114 7 months ago
5.0.112 112 7 months ago
5.0.111 132 7 months ago
5.0.110 123 7 months ago
5.0.109 122 7 months ago
5.0.108 126 8 months ago
5.0.107 124 8 months ago
5.0.106 103 8 months ago
5.0.105 85 8 months ago
5.0.104 100 8 months ago
5.0.103 102 8 months ago
5.0.102 119 9 months ago
5.0.101 114 9 months ago
5.0.100 121 9 months ago
5.0.99 100 9 months ago
5.0.98 131 9 months ago
5.0.97 118 10 months ago
5.0.96 127 10 months ago
5.0.95 125 10 months ago
5.0.94 109 10 months ago
5.0.93 108 10 months ago
5.0.92 118 5/31/2024
5.0.91 110 5/30/2024
5.0.90 112 5/17/2024
5.0.89 117 5/16/2024
5.0.88 137 5/8/2024
5.0.87 136 5/7/2024
5.0.86 143 5/6/2024
5.0.85 97 5/3/2024
5.0.84 106 5/2/2024
5.0.83 117 5/1/2024
5.0.82 135 4/30/2024
5.0.81 120 4/16/2024
5.0.80 120 4/12/2024
5.0.79 120 4/11/2024
5.0.78 141 4/1/2024
5.0.77 127 3/29/2024
5.0.76 131 3/18/2024
5.0.75 122 3/15/2024
5.0.74 126 3/14/2024
5.0.73 127 3/11/2024
5.0.72 118 3/8/2024
5.0.71 121 3/7/2024
5.0.70 141 3/6/2024
5.0.69 129 3/5/2024
5.0.68 132 3/4/2024
5.0.67 135 2/29/2024
5.0.66 119 2/28/2024
5.0.65 129 2/26/2024
5.0.64 129 2/23/2024
5.0.63 134 2/22/2024
5.0.62 138 2/21/2024
5.0.61 128 2/16/2024
5.0.60 129 2/15/2024
5.0.59 133 2/12/2024
5.0.58 124 2/8/2024
5.0.57 120 2/7/2024
5.0.56 121 2/6/2024
5.0.55 111 2/1/2024
5.0.54 125 1/31/2024
5.0.53 121 1/30/2024
5.0.52 115 1/24/2024
5.0.51 124 1/23/2024
5.0.50 129 1/12/2024
5.0.49 130 1/11/2024
5.0.48 137 12/26/2023
5.0.47 124 12/22/2023
5.0.46 111 12/18/2023
5.0.45 105 12/15/2023
5.0.44 102 12/14/2023
5.0.43 119 12/13/2023
5.0.42 112 12/12/2023
5.0.41 159 11/24/2023
5.0.40 138 11/21/2023
5.0.39 123 11/20/2023
5.0.38 128 11/17/2023
5.0.37 121 11/16/2023
5.0.36 127 11/14/2023
5.0.35 128 11/8/2023
5.0.34 112 11/7/2023
5.0.33 132 11/6/2023
5.0.32 128 11/1/2023
5.0.31 122 10/31/2023
5.0.30 139 10/30/2023
5.0.29 133 10/26/2023
5.0.28 157 10/12/2023
5.0.27 159 10/5/2023
5.0.26 147 9/26/2023
5.0.25 124 9/20/2023
5.0.24 119 9/19/2023
5.0.23 156 9/18/2023
5.0.22 135 9/14/2023
5.0.21 128 9/13/2023
5.0.20 142 9/11/2023
5.0.19 135 9/7/2023
5.0.18 136 9/6/2023
5.0.17 134 9/5/2023
5.0.16 151 9/4/2023
5.0.15 138 9/1/2023
5.0.14 146 8/31/2023
5.0.13 158 8/30/2023
5.0.12 158 8/29/2023
5.0.11 139 8/25/2023
5.0.10 150 8/23/2023
5.0.9 138 8/18/2023
5.0.8 163 8/10/2023
5.0.7 163 8/8/2023
5.0.6 169 8/8/2023
5.0.5 184 8/7/2023
5.0.4 168 8/3/2023
5.0.3 182 7/26/2023
5.0.2 161 7/20/2023
5.0.1 171 7/14/2023
5.0.0 325 12/12/2022
4.0.5 563 6/10/2022
4.0.2 523 1/11/2022
4.0.1 477 7/19/2021
3.0.9 533 1/7/2021
3.0.7 627 9/13/2020
3.0.6 591 6/26/2020
3.0.5 568 6/26/2020
3.0.3 584 3/25/2020
3.0.2 633 3/1/2020
3.0.1 651 1/1/2020
3.0.0 623 12/23/2019
2.0.15 658 11/22/2019
2.0.14 612 11/22/2019
2.0.13 594 11/22/2019
2.0.12 592 11/21/2019
2.0.11 586 11/21/2019
2.0.10 600 11/21/2019
2.0.9 587 11/21/2019
2.0.8 763 3/3/2019
2.0.7 693 3/3/2019
2.0.6 712 3/3/2019
2.0.5 695 3/3/2019
2.0.4 743 2/7/2019
2.0.3 1,118 6/1/2018
2.0.2 1,109 5/22/2018
2.0.1 1,241 1/2/2018
1.0.12 1,074 11/2/2017
1.0.11 1,071 10/30/2017
1.0.10 1,074 10/26/2017
1.0.9 1,073 10/26/2017
1.0.8 1,103 10/26/2017
1.0.7 1,056 10/25/2017
1.0.6 1,042 10/25/2017
1.0.5 1,078 10/24/2017
1.0.4 1,022 10/24/2017
1.0.3 1,014 10/19/2017
1.0.2 1,147 10/18/2017
1.0.1 1,068 9/29/2017