Category: HITS Scout

In the intro article we mentioned that HITS Scout’s link crawler is the feature that gets technical people’s attention. Here’s the full story, because the design is genuinely different from the usual “crawl everything at 2am” approach — and the difference matters if you care about your site’s load profile and how fresh your link data actually is.

The problem with the nightly full crawl

Traditional site checkers work like search-engine crawlers: they walk your entire site in one burst, compile a report, and go to sleep until tomorrow. That model has two failure modes. First, the burst itself — a crawler hitting every page on your site within a few minutes can look a lot like a slow-burn DoS, especially on modest shared hosting or a small VPS. Second, the staleness: a link that breaks at 9am gets reported tomorrow morning, after a full day of visitors finding it for you.

How the incremental crawler works

HITS Scout takes the opposite approach. Each monitor maintains a running inventory of every link on your domain — discovered as it encounters them, not in one upfront sweep. Then, every check cycle, alongside the regular uptime check of your primary URL, the crawler visits the oldest link it hasn’t looked at recently. One link per cycle. Steady, flat, boring — exactly what you want hitting a production site.

Think of it as a round-robin with a memory. Over time, every link in your inventory gets rechecked on a predictable cadence, and the effective coverage interval per link is simply the number of links multiplied by your monitor’s check interval. The math is worth doing for your own site, so here’s a worked example from ours: one of our monitors tracks roughly 26,000 links on a Gitea instance. At a 5-minute check interval, every link gets revisited about every 90 days. Drop the interval to 30 seconds and full coverage compresses to around 9 days. That’s the tuning dial: interval × link count = coverage cycle.

There’s a subtlety here worth appreciating: link importance is front-loaded automatically. Your homepage and primary navigation get checked every single cycle as part of uptime monitoring — the incremental crawl is spending its budget on the long tail of deep pages, which is precisely where breakage hides.

Same-domain only, and why that’s a feature

The crawler tracks same-domain links only. It won’t follow your outbound links to other people’s sites. That’s a deliberate boundary, and it’s the right one: your crawl stays bounded (the inventory can only grow as large as your own site), your crawler’s impact stays on infrastructure you own, and your link reports stay relevant — a broken outbound link on someone else’s site is interesting trivia, not your incident.

It also keeps the inventory honest. If the crawler wandered the open web, your “link count” metric would be meaningless — some fraction of it would be someone else’s problem. With same-domain tracking, the number of links in your inventory is a real measurement of your own site’s complexity, and you can watch it move as your site grows.

What each check records

Every link visit captures the HTTP status code and the response time, not just an up/down bit. That turns your link report into a performance dataset:

• Broken links — hard failures with status codes, so you can tell a dead page (404) from a server error (500) from a redirect that’s silently eating link equity (301 chains).
• Slow endpoints — pages that return 200 but take four seconds to do it. These never trigger a traditional uptime alert, but they’re exactly the pages that quietly bleed visitors and hurt your search rankings.
• Intermittent wobbles — the 503 that appears once an hour under load and vanishes before you can look. A single visit misses these; a rolling inventory with history catches the pattern.

Practical setup advice

A few rules of thumb we’ve settled on after running the crawler against everything from small marketing sites to that 26k-link Gitea:

• Match interval to site size. A 40-page marketing site at 5-minute checks gets full coverage in under a day and a half. A large documentation tree wants a faster interval if you want week-level coverage — or accept 90-day coverage and rely on alerting for the important pages.
• Watch the inventory number. If your link count suddenly jumps, either you shipped a big change or something is generating URLs (calendar pages, faceted search, session IDs in links). Both are worth knowing about.
• Fix in batches by depth. Broken-link reports are naturally ordered by when the crawler found them, which correlates with how deep in the site they are. Fix the shallow breakage first — it’s what visitors hit most.

The incremental crawler is included on every plan, including free — two monitors, full link inventories, no credit card. Point one at your biggest site and let the first report arrive: hitsscout.link/signup.

Next in this series: tuning check intervals and alert thresholds so you find real problems without paging yourself into oblivion.

Most uptime monitors tell you a comfortable lie. They ping your homepage every few minutes, get a 200 back, and paint your dashboard green. Meanwhile a CMS update quietly broke half your internal links, a plugin started 500-ing on a form submit, and the only people who know are your visitors — because they’re the ones hitting the 404s.

That gap is why we built HITS Scout. It’s a website monitoring platform that does both jobs properly: multi-region uptime monitoring and incremental link crawling across your whole site, with alerts that land where your team already lives.

The link crawler that doesn’t hammer your site

The feature that gets technical people’s attention is the crawler. HITS Scout doesn’t recrawl your entire site every night at 2am like a sitemap tool. It maintains a live inventory of every link on your domain and checks the oldest unvisited link each cycle. The result is a slow, steady crawl that spreads load across your check interval instead of dumping it in one burst — and it means link freshness is continuous, not a nightly report that’s stale by morning.

Every check records the status code and response time, so your link report isn’t just “broken or fine” — you can see the 301 chains, the slow endpoints, the intermittent 503s that only appear under load. And it tracks same-domain links only, which keeps crawls bounded to your site. Your crawler isn’t wandering off into third-party territory, and your link graph stays clean.

To give you a sense of scale: we dogfood HITS Scout on our own infrastructure, and one of our monitors currently tracks north of 26,000 links on a single Gitea instance. Incremental crawling is what makes that practical.

Multi-region uptime, not single-datacenter optimism

A monitor in one region tells you your site is up from that region. That’s a meaningful difference, and anyone who’s debugged a “works for me” incident knows it. HITS Scout runs distributed monitoring agents across multiple regions, so a regional DNS hiccup, a CDN edge problem, or a routing weirdness shows up as what it is — a regional issue — rather than either being missed entirely or paging you as a full outage.

Check intervals are configurable per monitor, from 30 seconds up to 24 hours. You can watch your production app like a hawk while checking the marketing site every few hours. Each monitor tracks uptime percentage and full downtime history, so when the client asks “was it down last Tuesday?”, you have the answer instead of a shrug.

Alerts that don’t cry wolf

Alert fatigue kills monitoring setups faster than any outage. HITS Scout’s alerting has a verification step built in: before it alerts on link-check failures, it verifies the primary URL first. If your whole site is down, you get one clear alert, not forty. If the homepage is fine but deep links are broken, you get the link report — a different signal, treated differently. Thresholds are configurable so you can tune sensitivity per monitor.

Notifications go out via Discord, Slack, or email — whichever channel your team actually watches. No proprietary app to install, no vendor portal to log into at 2am.

Status pages and the activity timeline

Every monitor can feed a public status page — a curated, customer-facing view with live uptime, response times, and recent incidents. Viewers don’t need an account, which is the entire point of a status page. If you run client sites, this is the difference between “we’re looking into it” and a link you can paste into an email.

Behind the scenes, the activity timeline gives you a unified feed across all your monitors: outages, slow responses, and link issues in one chronological stream. When three sites on the same host start acting up within the same ten-minute window, the timeline makes the correlation obvious instead of something you reconstruct from three separate dashboards.

Built like a proper distributed system

For the infra-curious: HITS Scout is a Go backend with small, single-purpose workers — an alert worker, usage worker, metrics worker, pruner, and agent cleanup — coordinated through a queue, with monitoring agents as an independent tier that can be scaled by region. The capacity system is built on measured data: real checks-per-minute throughput, per-agent utilisation, and queue depth, so the platform’s own health dashboard tells you the same kind of truth it tells you about your websites. The frontend is React and TypeScript, and the whole stack deploys cleanly on Docker Compose or Kubernetes.

We’re a DevOps consultancy, and this architecture is the one we’d want to inherit: boring components, clear separation, no distributed monolith pretending to be microservices.

Pricing that fits a homelab budget

The free tier gives you 2 monitors with 5-minute checks and 7 days of retention — genuinely free, no credit card. Pro is $4/month for 5 monitors at 1-minute checks and 30 days retention, and Enterprise is $9/month for 10 monitors at 30-second checks and 90 days retention. That’s cheap enough to point monitors at every side project, every client site, and the stuff you’re not paid enough to care about but would still feel bad losing.

Start monitoring in about two minutes

Sign up at hitsscout.link/signup, add a URL, and HITS Scout starts crawling and uptime-checking immediately. Your first broken-link report will probably arrive before you’ve finished your coffee — most sites have at least a few skeletons in the link closet.

We’ll be writing follow-up articles here covering specific features in depth: tuning check intervals, setting up alert channels, building a public status page, and getting the most out of the link reports. If you’ve got questions in the meantime, the docs and signup live at hitsscout.com.