Web Scraping With C# – Tools and Patterns

Informational P2 web scraping with c#

Learn how to scrape in C#: HttpClient done correctly, AngleSharp for parsing, bounded async concurrency, and Playwright for rendered pages.

The .NET ecosystem covers collection well: a capable HTTP client, a standards-compliant HTML parser, first-class async, and official Playwright bindings for rendering.

1. Use HttpClient Correctly

// one instance, reused — or better, IHttpClientFactory in hosted apps
static readonly HttpClient Client = new(new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(5),
    MaxConnectionsPerServer = 8,
})
{
    Timeout = TimeSpan.FromSeconds(30),
};

using var response = await Client.GetAsync(url, cancellationToken);
response.EnsureSuccessStatusCode();
var html = await response.Content.ReadAsStringAsync(cancellationToken);

Creating a client per request exhausts sockets — the best-known pitfall in .NET HTTP code. One shared instance with a pooled connection lifetime avoids both that and stale DNS.

2. Parsing With AngleSharp

var context = BrowsingContext.New(Configuration.Default);
var document = await context.OpenAsync(req => req.Content(html).Address(url));

foreach (var card in document.QuerySelectorAll("article.product"))
{
    var title = card.QuerySelector("h3")?.TextContent.Trim();
    var price = card.QuerySelector(".price")?.TextContent.Trim();
}

AngleSharp implements the same parsing and selector standards a browser follows, so its tree matches what a browser would build — including its repairs to malformed markup.

3. Bounded Concurrency

using var gate = new SemaphoreSlim(8);
var tasks = urls.Select(async url =>
{
    await gate.WaitAsync(cancellationToken);
    try { return await CollectAsync(url, cancellationToken); }
    finally { gate.Release(); }
});
var results = await Task.WhenAll(tasks);

MaxConnectionsPerServer bounds connections; the semaphore bounds tasks in flight. Both are worth setting.

4. Cancellation Throughout

Passing a CancellationToken through every async call means a collection can be stopped promptly rather than waiting for every in-flight request to time out. It is also how a graceful shutdown is implemented in a hosted service.

5. Rendered Pages

Playwright has official .NET bindings, so client-rendered pages are handled with the same API available in other languages:

await using var browser = await playwright.Chromium.LaunchAsync();
var page = await browser.NewPageAsync();
await page.GotoAsync(url, new() { WaitUntil = WaitUntilState.DOMContentLoaded });
await page.WaitForSelectorAsync("article.product");

Check for the page's underlying JSON endpoint first; a browser costs far more per page.

6. Typed Output

Records give a concise, immutable shape for extracted data, and System.Text.Json serialises them to JSON Lines. Use decimal for money rather than double, and DateTimeOffset for timestamps so time zone information survives.

7. Practical Notes

One HttpClient, reused, with a pooled connection lifetime.
Explicit timeouts and cancellation tokens everywhere.
Null-conditional access on selectors; elements go missing.
Per-host connection limits rather than only a global cap.
Stream output instead of accumulating large collections.

8. Summary

C# collects with a correctly reused HttpClient, AngleSharp for standards-compliant parsing, semaphore-bounded async concurrency, and Playwright when rendering is required.

The pitfalls are ecosystem-specific rather than conceptual: client lifetime, cancellation propagation, and choosing decimal types for money.