PageProbe 1.0.7

PageProbe

PageProbe is a .NET-based web crawling library designed to monitor and extract content from statically generated websites. It enables developers, IT students, and enthusiasts to gather and export data such as links, media, metadata, multimedia, text, or even price information. The library supports both real-time and scheduled crawling, with the ability to store results and compare differences over time.

Features

• Modular and extensible architecture using interfaces and models
• Extract links, images, videos, metadata, and plain text from webpages
• Specialized price crawler with regex-based rule support
• Export to multiple formats (JSON, CSV, XML, TXT, Markdown)
• Snapshot comparisons for detecting changes
• Console and file-based output
• Asynchronous support for crawling over time

Installation

1. NuGet Package (Recommended)

Install-Package PageProbe

2. Clone Repository

git clone https://dev.azure.com/emilberglund/_git/rammeverk_gruppe2

Then add the project to your solution and reference it in your code:

using PageProbe;

.NET Requirement: Ensure you are using .NET 8.0 or newer.

Project Structure

Main Components

🕷 Crawlers

• BaseCrawler: Core logic for HTML parsing and extraction
• PriceCrawler: Extends BaseCrawler, specialized for extracting price data using defined regex rules

🧩 Interfaces

• ILinkCrawler: Link extraction
• IMediaCrawler: Image/media extraction
• IMetadataCrawler: Meta tags (title, description, keywords)
• IMultimediaCrawler: Video/audio/iframe extraction
• ITextCrawler: Plain text extraction
• IExporter, IExportDifferences: Export results and diffs

📦 Models

• CrawlResult: Snapshot of crawled content
• CrawlDifferences: Differences between crawl snapshots
• PriceExtractionRule: Regex-based rule for price detection

📁 File Handlers (Exporters)

• JsonExporter, CsvExporter, MarkdownExporter, TextExporter, XmlExporter: Export to corresponding file formats

Basic Usage

Crawl a URL and Export the Results

var crawler = new BaseCrawler();
var url = "https://example.no/";

var result = new CrawlResult
{
    Url = url,
    Title = crawler.GetMetaTitle(url),
    Description = crawler.GetMetaDescription(url),
    Keywords = crawler.GetMetaKeywords(url).Split(',', StringSplitOptions.RemoveEmptyEntries).ToList(),
    Links = crawler.GetLinks(url),
    Images = crawler.GetImages(url),
    Multimedia = crawler.GetMultimedia(url),
    Text = crawler.GetText(url)
};

var exporter = new JsonExporter();
exporter.Export(result, "results");

Export Asynchronously

await exporter.ExportAsync(result, "results-async");

Show Links in Console

crawler.GetLinks(url).ToList().ForEach(Console.WriteLine);

Full Documentation

See the full documentation here for architecture, API reference, and advanced usage.

Build & Test

Build the Solution

1. Open in Visual Studio
2. Press Ctrl + Shift + B

Run Tests

dotnet test

Contributing

As this is an internal project:

• Use feature/* branches
• Follow .NET conventions
• Write meaningful commit messages

License

This project is for internal use only and not distributed publicly.