Web Crawler

A focused demo of WebSpark.HttpClientUtility.Crawler — the companion NuGet package that adds site crawling to the base HTTP utility.

Run a Crawl
Crawls only same-domain links. SPAs (React/Vue/Angular) are not supported — use server-rendered sites.
Demo cap: 25 pages
Package Setup

Install both packages:

dotnet add package WebSpark.HttpClientUtility
dotnet add package WebSpark.HttpClientUtility.Crawler

Register in Program.cs:

services.AddHttpClientUtility();
services.AddHttpClientCrawler();
Crawler Features
  • robots.txt compliance
    Reads and respects Disallow rules before crawling any path
  • Sitemap & RSS discovery
    Auto-seeds the queue from sitemap.xml, rss.xml, atom.xml
  • HTML link extraction
    Parses <a href> tags via HtmlAgilityPack, same-domain only
  • Depth-limited crawling
    BFS with configurable MaxDepth and MaxPages guards
  • Adaptive rate limiting
    Adjusts delays based on server response times to avoid overloading hosts
  • Concurrent requests
    Configurable MaxConcurrentRequests for throughput control
Usage Example
var result = await crawler.CrawlAsync(
    "https://markhazleton.com",
    new CrawlerOptions {
        MaxPages = 20,
        MaxDepth = 2,
        RespectRobotsTxt = true,
        DiscoverFromSitemapAndRss = true,
        RequestDelayMs = 300
    });