The project is SiteHealth, an AI-powered website health monitor. The idea: point it at a website, and it finds broken links, SEO problems, slow pages and accessibility issues, then uses Claude to explain each problem and suggest a fix. Eventually it will run on Azure with queues, workers and a React dashboard, and show results inside the Optimizely CMS editor.

But every big project starts small. Day 1 was a command-line link checker.

🔗 Code: github.com/jasdeep-singh2/sitehealth

What it does

dotnet run --project src/SiteHealth.Cli -- https://books.toscrape.com/

Fetching https://books.toscrape.com/ ...
Found 73 links. Checking them...

0 broken out of 73 links.

It works in three steps:

1. PageFetcher downloads the page's HTML.
2. LinkExtractor finds every link on the page.
3. LinkChecker checks each link, five at a time, and reports the broken ones.

I wrote the tests first and then made them pass one step at a time. By the end of the day: 15 tests, all green.

Here are the three things I learned that I think are worth sharing.

Lesson 1: never new HttpClient() per request

This is one of the classic .NET mistakes:

// ❌ Don't do this
using var client = new HttpClient();
var html = await client.GetStringAsync(url);

Creating a new HttpClient for every request leaves sockets hanging around after it's disposed. Under load, you can run out of them, which is known as socket exhaustion.

The fix is IHttpClientFactory. I register a typed client, and the class simply asks for an HttpClient in its constructor:

builder.Services.AddHttpClient<PageFetcher>(ConfigureCrawlerClient);

public sealed class PageFetcher(HttpClient httpClient)
{
    public async Task<string> FetchHtmlAsync(Uri url, CancellationToken cancellationToken)
    {
        using var response = await httpClient.GetAsync(url, cancellationToken);
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync(cancellationToken);
    }
}

The factory reuses connections behind the scenes, and there's a bonus I didn't expect: testing becomes easy. Because PageFetcher receives its HttpClient instead of creating it, my tests pass in an HttpClient backed by a fake handler. The tests never touch the real internet, but PageFetcher can't tell the difference.

var handler = new StubHttpMessageHandler(_ => new HttpResponseMessage(HttpStatusCode.NotFound));
var fetcher = new PageFetcher(new HttpClient(handler));

await Assert.ThrowsAsync<HttpRequestException>(() => fetcher.FetchHtmlAsync(url, CancellationToken.None));

Lesson 2: check links in parallel, but politely

Checking 73 links one by one is slow. Checking all 73 at once hammers the website, which is rude and can get you blocked.

Parallel.ForEachAsync with a limit gives the best of both:

var parallelOptions = new ParallelOptions
{
    MaxDegreeOfParallelism = _options.MaxConcurrency, // 5, from appsettings.json
    CancellationToken = cancellationToken,
};

await Parallel.ForEachAsync(links, parallelOptions, async (url, token) =>
{
    results.Add(await CheckOneAsync(url, token));
});

Two details I learned along the way:

• ConcurrentBag instead of List: five checks run at the same time, and a normal List isn't safe to add to from several threads.
• HEAD before GET: a HEAD request asks "does this page exist?" without downloading it. Some servers refuse HEAD with 405 Method Not Allowed, so the checker falls back to GET.

This limit is my first real system design concept: back-pressure, controlling how much work runs at once so neither side gets overwhelmed. Later in the project, Azure Service Bus and scaled-out workers will do the same job at a bigger scale.

Lesson 3: a timeout and Ctrl+C throw the same exception

This one surprised me. When a website is too slow, HttpClient throws a TaskCanceledException. When the user presses Ctrl+C and the cancellation token fires, it also throws a TaskCanceledException.

But they need different handling:

• Timeout: the site is slow, so record the link as broken and carry on.
• Ctrl+C: the user wants to stop, so actually stop.

An exception filter tells them apart:

catch (TaskCanceledException) when (!cancellationToken.IsCancellationRequested)
{
    // Our token wasn't cancelled, so this was a timeout
    return new LinkCheckResult(url, null, "Timed out");
}

If the user cancelled, the filter doesn't match, and the exception bubbles up and stops the program, which is exactly what we want.

How I'm using AI to learn (not to skip learning)

I'm using Claude as a tutor rather than a code generator. It set up the project skeleton and the tests, then gave me fill-in-the-blanks exercises for each step. The tests defined when I was "done," and I typed and debugged every implementation myself.

That balance has worked well: I moved fast, but I understand every line I committed.

What's next

Step 4: send the broken links to Claude and get back a plain-English explanation and a suggested fix for each one, as structured JSON. That will be my first real AI integration in .NET.

After that: moving the checker into a background worker, then onto Azure with a queue.

If you're a .NET or Optimizely developer learning AI or Azure too, I'd love to hear what you're building. The code is all on GitHub