RafaelEstevam.Simple.Spider.SqliteStorage 0.8.2

Simple.Spider

A simple and modular web spider written in C# .Net

• Core 3.1
• Net 5.0

This Package was moved from [Net.RafaelEstevam.Spider.Simple.Lib]

Content

• Simple.Spider
• Content
  • Some advantages
  • Getting started
  • Samples

Some advantages

• Very simple to use and operate, ideal for lots of small projects or personal ones
• Easy html filter with HObject (a HtmlNode wrap with use similar to JObject)
• Internal conversion from html to XElement, no need to external tools on use
• Automatic Json parser to JObject
• Automatic Json deserialize <T>
• Modular Parser engine (you can add your own parsers!)
  • JSON and XML already included
• Modular Caching engine (you can add your own!)
  • Stand alone Cache engine included, no need to external softwares
• Modular Downloader engine (you can add your own!)
  • WebClient with cookies or HttpClient download engine included

Easy import with NuGet

Getting started

1. Start a new console project and add Nuget Reference
2. PM> Install-Package Net.RafaelEstevam.Spider.Simple.Lib
3. Create a class for your spider (or leave in program)
4. create a new instance of SimpleSpider
   1. Give it a name, cache and log will be saved with that name
   2. Give it a domain (your spider will not fleet from it)
5. Add a event FetchCompleted to
6. Optionally give a first page with AddPage. If omitted, it will use the home page of the domain
7. Call Execute()

void run()
{
    var spider = new SimpleSpider("QuotesToScrape", new Uri("http://quotes.toscrape.com/"));
    // Set the completed event to implement your stuff
    spider.FetchCompleted += fetchCompleted_items;
    // execute
    spider.Execute();
}
void fetchCompleted_items(object Sender, FetchCompleteEventArgs args)
{
    // walk around ...
    // TIP: inspect args to see stuff

    var hObj = args.GetHObject();
    string[] quotes = hObj["span > .text"];
}

TIP: Use the Simple.Tests project to see examples and poke around

Samples

See all samples at Simple.Tests