tsllbn
-
- 55,621 total downloads
- last updated 2/19/2014
- Latest version: 2014.2.19
A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards. See https://bitbucket.org/cagdas/robotstxt for usage examples.