SuperSpider 1.0.4

This package has been deprecated as it has critical bugs.

There is a newer version of this package available.
See the version list below for details.

dotnet add package SuperSpider --version 1.0.4

NuGet\Install-Package SuperSpider -Version 1.0.4

This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.

<PackageReference Include="SuperSpider" Version="1.0.4" />

For projects that support PackageReference, copy this XML node into the project file to reference the package.

paket add SuperSpider --version 1.0.4

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

#r "nuget: SuperSpider, 1.0.4"

#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.

// Install SuperSpider as a Cake Addin
#addin nuget:?package=SuperSpider&version=1.0.4

// Install SuperSpider as a Cake Tool
#tool nuget:?package=SuperSpider&version=1.0.4

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

SuperSpider

Crawler Framework For .NET

Support .NET 8.0

功能特点

1、CodeFirst，根据实体和配置文件，自动建库建表，实体需继承自SpiderEntity

2、UseStorage()，支持数据持久化，自动新增或更新，可定制化

3、支持XPath和JsonPath两种解析方式

4、更加轻量级，源码仅仅 27 KB

5、使用简单、快速开发

6、更多功能，敬请期待

安装

SuperSpider 以 NuGet 包的形式提供。您可以使用 NuGet 包控制台窗口安装它：

PM> Install-Package SuperSpider

使用

Program.cs

using SuperSpider;

using var request_wb = new Request<WeiboEntity>
{
    RequestUri = new Uri("https://s.weibo.com/top/summary?cate=realtimehot"),
    Headers =
    {
        { HeaderNames.Cookie, "SUB=_2AkMRPey0f8NxqwFRmP0QzG7jZIh-zA_EieKnYR1vJRMyHRl-yD9yqhMPtRB6Or3CWw-34jkWWR4Y0x2HL1v5PpcCYaf4" },
        { HeaderNames.UserAgent, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" }
    },
    FilterConditions = x => x.RankTop.HasValue,
    DuplicateColumns = x => new { x.Keywords }
};

Request[] requests = [request_wb];

await EntitySpider
    .Build(requests)
    .UseStorage()
    .RunAsync();

var results = request_wb.Results;

WeiboEntity.cs

[SpiderSchema("weibo", "微博热搜")]
[SpiderIndex("unique_weibo_Keywords", nameof(Keywords), true, true)]
[EntitySelector(Expression = "//*[@id=\"pl_top_realtimehot\"]/table/tbody/tr")]
public class WeiboEntity : SpiderEntity
{
    [SpiderColumn(ColumnDescription = "排行", IsNullable = true)]
    [RegexFormatter(Pattern = "[0-9]+")]
    [ValueSelector(Expression = "//td[1]")]
    public int? RankTop { get; set; }

    [SpiderColumn(ColumnDescription = "关键词", IsNullable = true)]
    [ValueSelector(Expression = "//td[2]/a")]
    public string? Keywords { get; set; }

    [SpiderColumn(ColumnDescription = "热度文本", IsNullable = true)]
    [RegexFormatter(Pattern = "[\u4E00-\u9FA5]+")]
    [ValueSelector(Expression = "//td[2]/span")]
    public string? HotText { get; set; }

    [SpiderColumn(ColumnDescription = "热度值", IsNullable = true)]
    [RegexFormatter(Pattern = "[0-9]+")]
    [ValueSelector(Expression = "//td[2]/span")]
    public int? HotValue { get; set; }

    [SpiderColumn(ColumnDescription = "热度标签", IsNullable = true)]
    [ValueSelector(Expression = "//td[3]/i")]
    public string? HotTag { get; set; }

    [SpiderColumn(ColumnDescription = "链接", IsNullable = true)]
    [ValueSelector(Expression = "//td[2]/a/@href")]
    public string? Url { get; set; }
}

appsettings.json

{
  "SpiderConfig": {
    "ConnectionConfig": [
      {
        "ConfigId": 1,
        "DbType": 0,
        "InitKeyType": 1,
        "IsAutoCloseConnection": true,
        "ConnectionString": "server=127.0.0.1;port=3306;uid=root;pwd=123456;database=hotsearch;max pool size=8000;charset=utf8;",
        "SlaveConnectionConfigs": [
          {
            "ConnectionString": "server=127.0.0.1;port=3306;uid=root;pwd=123456;database=hotsearch;max pool size=8000;charset=utf8;",
            "HitRate": 10
          }
        ]
      }
    ]
  }
}

Product	Compatible and additional computed target framework versions.
.NET	net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed.