ParkSquare.BotProtection 1.0.3

ParkSquare.BotProtection

ASP.NET Core middleware that slows down unwanted bots (including AI bots) and blocks IP addresses that keep producing bad responses.

Almost as soon as a public-facing website or API goes live, bots start to fire lots of requests at it - mostly for files and pages that do not exist. This is designed to probe your website or API to see what technology is used, and whether anything has been left exposed that could be exploited.

Often, you will see requests for random PHP files (even if your site is built in C#), configuration files, and known Wordpress stuff.

In the absence of a WAF (web application firewall), this package will block a bot's IP address for a configurable amount of time after a set number of failed requests.

It can also be used to whitelist specific bots and block others, for example you may wish to allow Googlebot while blocking AI bots that are scraping your site.

It does two things:

1. Bot tarpit Requests from a recognised bot (detected from the User-Agent with DeviceDetector.NET) that is not on your allow list are held for a set delay, then answered with 404 Not Found. This wastes the bot's time, and slows them down. Allowed bots, such as search engine crawlers, pass straight through. A cap on concurrent tarpits stops the middleware itself being used to exhaust threads or memory: when the cap is reached, the request gets an immediate 404.
2. Bad request blocker Each client IP is counted when it gets a 400, 401, 403 or 404 (or a tarpitted request). Once an IP exceeds the strike limit inside the window, it is blocked for a period and receives 429 Too Many Requests.

Getting Started

Install the package:

dotnet add package ParkSquare.BotProtection

Register the services in Program.cs (or ConfigureServices):

builder.Services.AddBotProtection();

BotProtectionConfig binds the BotProtection section of appsettings.json. Any setting you leave out uses its default, listed in Options:

{
  "BotProtection": {
    "Tarpit": {
      "Delay": "00:00:10",
      "MaxConcurrent": 32
    },
    "BadRequest": {
      "Window": "00:30:00",
      "BlockFor": "00:15:00",
      "MaxBadResponses": 5,
      "CountedStatusCodes": [ 400, 401, 403, 404 ]
    },
    "AllowedBots": [ "Google" ],
    "ExemptPaths": [ "/robots.txt" ],
    "WhitelistedIps": [ "10.0.0.0/8", "203.0.113.5" ]
  }
}

Add the middleware. It must run after UseForwardedHeaders, because the blocker keys on the client IP that middleware resolves:

app.UseForwardedHeaders(forwardedHeadersOptions);
app.UseExceptionHandler("/error/500");
app.UseStatusCodePagesWithReExecute("/error/{0}");

app.UseBotProtection();

Put UseBotProtection() ahead of UseStaticFiles() if you want bots blocked from static content as well.

Options

Any setting left out of appsettings.json, or set to zero or empty, uses the default below. AllowedBots and ExemptPaths default to empty.

Allowed bots are also exempt from the bad request blocker: they are neither counted nor blocked.

State Persistence

The default store uses IMemoryCache. To share strikes and blocks across container replicas, implement IBadRequestStore and register it before calling AddBotProtection():

builder.Services.AddSingleton<IBadRequestStore, RedisBadRequestStore>();
builder.Services.AddBotProtection(...);

Only the default IMemoryCache store ships with this package.