Posts

Posted on

How HITS Scout’s incremental link crawler works

In the intro article we mentioned that HITS Scout’s link crawler is the feature that gets technical people’s attention. Here’s the full story, because the design is genuinely different from the usual “crawl everything at 2am” approach — and the difference matters if you care about your site’s load profile and how fresh your link data actually is.

The problem with the nightly full crawl

Traditional site checkers work like search-engine crawlers: they walk your entire site in one burst, compile a report, and go to sleep until tomorrow. That model has two failure modes. First, the burst itself — a crawler hitting every page on your site within a few minutes can look a lot like a slow-burn DoS, especially on modest shared hosting or a small VPS. Second, the staleness: a link that breaks at 9am gets reported tomorrow morning, after a full day of visitors finding it for you.

How the incremental crawler works

HITS Scout takes the opposite approach. Each monitor maintains a running inventory of every link on your domain — discovered as it encounters them, not in one upfront sweep. Then, every check cycle, alongside the regular uptime check of your primary URL, the crawler visits the oldest link it hasn’t looked at recently. One link per cycle. Steady, flat, boring — exactly what you want hitting a production site.

Think of it as a round-robin with a memory. Over time, every link in your inventory gets rechecked on a predictable cadence, and the effective coverage interval per link is simply the number of links multiplied by your monitor’s check interval. The math is worth doing for your own site, so here’s a worked example from ours: one of our monitors tracks roughly 26,000 links on a Gitea instance. At a 5-minute check interval, every link gets revisited about every 90 days. Drop the interval to 30 seconds and full coverage compresses to around 9 days. That’s the tuning dial: interval × link count = coverage cycle.

There’s a subtlety here worth appreciating: link importance is front-loaded automatically. Your homepage and primary navigation get checked every single cycle as part of uptime monitoring — the incremental crawl is spending its budget on the long tail of deep pages, which is precisely where breakage hides.

Same-domain only, and why that’s a feature

The crawler tracks same-domain links only. It won’t follow your outbound links to other people’s sites. That’s a deliberate boundary, and it’s the right one: your crawl stays bounded (the inventory can only grow as large as your own site), your crawler’s impact stays on infrastructure you own, and your link reports stay relevant — a broken outbound link on someone else’s site is interesting trivia, not your incident.

It also keeps the inventory honest. If the crawler wandered the open web, your “link count” metric would be meaningless — some fraction of it would be someone else’s problem. With same-domain tracking, the number of links in your inventory is a real measurement of your own site’s complexity, and you can watch it move as your site grows.

What each check records

Every link visit captures the HTTP status code and the response time, not just an up/down bit. That turns your link report into a performance dataset:

  • Broken links — hard failures with status codes, so you can tell a dead page (404) from a server error (500) from a redirect that’s silently eating link equity (301 chains).
  • Slow endpoints — pages that return 200 but take four seconds to do it. These never trigger a traditional uptime alert, but they’re exactly the pages that quietly bleed visitors and hurt your search rankings.
  • Intermittent wobbles — the 503 that appears once an hour under load and vanishes before you can look. A single visit misses these; a rolling inventory with history catches the pattern.

Practical setup advice

A few rules of thumb we’ve settled on after running the crawler against everything from small marketing sites to that 26k-link Gitea:

  • Match interval to site size. A 40-page marketing site at 5-minute checks gets full coverage in under a day and a half. A large documentation tree wants a faster interval if you want week-level coverage — or accept 90-day coverage and rely on alerting for the important pages.
  • Watch the inventory number. If your link count suddenly jumps, either you shipped a big change or something is generating URLs (calendar pages, faceted search, session IDs in links). Both are worth knowing about.
  • Fix in batches by depth. Broken-link reports are naturally ordered by when the crawler found them, which correlates with how deep in the site they are. Fix the shallow breakage first — it’s what visitors hit most.

The incremental crawler is included on every plan, including free — two monitors, full link inventories, no credit card. Point one at your biggest site and let the first report arrive: hitsscout.link/signup.

Next in this series: tuning check intervals and alert thresholds so you find real problems without paging yourself into oblivion.

Posted on

Introducing HITS Scout: uptime monitoring that checks more than your homepage

Most uptime monitors tell you a comfortable lie. They ping your homepage every few minutes, get a 200 back, and paint your dashboard green. Meanwhile a CMS update quietly broke half your internal links, a plugin started 500-ing on a form submit, and the only people who know are your visitors — because they’re the ones hitting the 404s.

That gap is why we built HITS Scout. It’s a website monitoring platform that does both jobs properly: multi-region uptime monitoring and incremental link crawling across your whole site, with alerts that land where your team already lives.

The link crawler that doesn’t hammer your site

The feature that gets technical people’s attention is the crawler. HITS Scout doesn’t recrawl your entire site every night at 2am like a sitemap tool. It maintains a live inventory of every link on your domain and checks the oldest unvisited link each cycle. The result is a slow, steady crawl that spreads load across your check interval instead of dumping it in one burst — and it means link freshness is continuous, not a nightly report that’s stale by morning.

Every check records the status code and response time, so your link report isn’t just “broken or fine” — you can see the 301 chains, the slow endpoints, the intermittent 503s that only appear under load. And it tracks same-domain links only, which keeps crawls bounded to your site. Your crawler isn’t wandering off into third-party territory, and your link graph stays clean.

To give you a sense of scale: we dogfood HITS Scout on our own infrastructure, and one of our monitors currently tracks north of 26,000 links on a single Gitea instance. Incremental crawling is what makes that practical.

Multi-region uptime, not single-datacenter optimism

A monitor in one region tells you your site is up from that region. That’s a meaningful difference, and anyone who’s debugged a “works for me” incident knows it. HITS Scout runs distributed monitoring agents across multiple regions, so a regional DNS hiccup, a CDN edge problem, or a routing weirdness shows up as what it is — a regional issue — rather than either being missed entirely or paging you as a full outage.

Check intervals are configurable per monitor, from 30 seconds up to 24 hours. You can watch your production app like a hawk while checking the marketing site every few hours. Each monitor tracks uptime percentage and full downtime history, so when the client asks “was it down last Tuesday?”, you have the answer instead of a shrug.

Alerts that don’t cry wolf

Alert fatigue kills monitoring setups faster than any outage. HITS Scout’s alerting has a verification step built in: before it alerts on link-check failures, it verifies the primary URL first. If your whole site is down, you get one clear alert, not forty. If the homepage is fine but deep links are broken, you get the link report — a different signal, treated differently. Thresholds are configurable so you can tune sensitivity per monitor.

Notifications go out via Discord, Slack, or email — whichever channel your team actually watches. No proprietary app to install, no vendor portal to log into at 2am.

Status pages and the activity timeline

Every monitor can feed a public status page — a curated, customer-facing view with live uptime, response times, and recent incidents. Viewers don’t need an account, which is the entire point of a status page. If you run client sites, this is the difference between “we’re looking into it” and a link you can paste into an email.

Behind the scenes, the activity timeline gives you a unified feed across all your monitors: outages, slow responses, and link issues in one chronological stream. When three sites on the same host start acting up within the same ten-minute window, the timeline makes the correlation obvious instead of something you reconstruct from three separate dashboards.

Built like a proper distributed system

For the infra-curious: HITS Scout is a Go backend with small, single-purpose workers — an alert worker, usage worker, metrics worker, pruner, and agent cleanup — coordinated through a queue, with monitoring agents as an independent tier that can be scaled by region. The capacity system is built on measured data: real checks-per-minute throughput, per-agent utilisation, and queue depth, so the platform’s own health dashboard tells you the same kind of truth it tells you about your websites. The frontend is React and TypeScript, and the whole stack deploys cleanly on Docker Compose or Kubernetes.

We’re a DevOps consultancy, and this architecture is the one we’d want to inherit: boring components, clear separation, no distributed monolith pretending to be microservices.

Pricing that fits a homelab budget

The free tier gives you 2 monitors with 5-minute checks and 7 days of retention — genuinely free, no credit card. Pro is $4/month for 5 monitors at 1-minute checks and 30 days retention, and Enterprise is $9/month for 10 monitors at 30-second checks and 90 days retention. That’s cheap enough to point monitors at every side project, every client site, and the stuff you’re not paid enough to care about but would still feel bad losing.

Start monitoring in about two minutes

Sign up at hitsscout.link/signup, add a URL, and HITS Scout starts crawling and uptime-checking immediately. Your first broken-link report will probably arrive before you’ve finished your coffee — most sites have at least a few skeletons in the link closet.

We’ll be writing follow-up articles here covering specific features in depth: tuning check intervals, setting up alert channels, building a public status page, and getting the most out of the link reports. If you’ve got questions in the meantime, the docs and signup live at hitsscout.com.

Posted on

God’s Eye View: an open-source spy-globe in your browser

Every so often an open-source project lands that feels less like software and more like a magic trick. God’s Eye View is a browser-based “spy satellite simulator” — a photorealistic 3D globe covered in live data: aircraft transponders, ship beacons, orbital elements for satellites, earthquakes, public traffic cameras. Click an aircraft and the camera locks on, drawing a fading trail. Ride along in a cockpit view that holds the terrain all the way down. Ask it, out loud, to show you something. It hit #1 on GitHub Trending in August, and one look at the demo explains why.

The project grew out of the God’s Eye View video series (5M+ views on YouTube), and the interesting part isn’t the drama — it’s the plumbing. Everything on the globe comes from public, mostly open feeds: ADS-B flight data, AIS ship positions, seismic networks, broadcast orbital elements. What the project does is fuse them into one explorable picture, running locally in your browser, with source you can read. As the README puts it: half the magic is that it looks like a forbidden cockpit, and the other half is that every line of code is inspectable.

Getting started is unusually friendly for something this elaborate. It runs without any API keys — flights, satellites, earthquakes and cameras work out of the box — and can be installed with one click through Pinokio, or from a terminal with the standard npm ci && npm run dev. Optional keys (Cesium ion for photorealistic terrain, Google Maps for search) are upgrades, not prerequisites, added through an in-app panel. The README is refreshingly honest about what’s real versus estimated: traffic is simulated along real roads, and camera positions are coarse estimates.

For home lab types, there are two hooks. First, it’s a genuinely fun self-hosted showpiece — runs on your own hardware, no accounts, no telemetry middleman. Second, it’s a masterclass in layering open data sources, and each layer is a separate module: if you have an idea for your own feed (a local sensor, a fleet tracker, anything with coordinates), it’s designed to be extended rather than just admired.

Worth a look even if you never extend it. There’s something quietly reassuring about a project that takes the surveillance aesthetic everyone finds unsettling and builds it entirely from data anyone can download — the trick turned inside out, handed back to you with the source attached.

Posted on

Cheap ESP32 touch screens for Home Assistant, without a line of YAML

Ask anyone running Home Assistant what they want more of, and the answer is usually “screens”. Your phone is in the other room, a wall-mounted tablet costs real money, and smart speakers only go so far. A project called ESP Screen attacks this from the other end: it puts full touch control on cheap ESP32 panels — the 2.8-inch “CYD” boards that sell for pocket change, and the 4-inch Guition — with a drag-and-drop editor and, remarkably, not a single line of YAML.

That last part is the headline. The usual ESP32 dashboard route means firmware flashing, MQTT brokers, tokens and hand-written layouts. ESP Screen instead installs as a manager inside Home Assistant itself: it flashes a new panel over USB, updates it over Wi-Fi afterwards, and pushes your layout to it. Designing a screen is pick a tile, drag it, save — lights, climate, blinds, the vacuum robot, media players, weather, history graphs, clocks and timers, up to 48 tiles across eight pages. On the Guition, cameras work too: the doorbell’s last snapshot as a tile, full-screen on a tap, refreshed every few seconds.

The polish is what makes it feel finished rather than a hobby project. Tap toggles, hold opens the full control card. A lamp with effects gets an effects picker filled in from what Home Assistant actually exposes. Dark mode suits a bedside screen, and night-hours dimming can be driven by an automation — as can an alert that pops on every screen in the house when someone rings the bell, complete with the camera picture on the Guition. Custom YAML for one screen even survives firmware updates, a small courtesy that tells you the author has lived with this daily.

A few honest caveats from the README’s own fine print: the CYD’s memory is too small for camera pictures and album art (it substitutes icons), so treat the 4-inch Guition as the one to buy if cameras matter. Panels this cheap mean modest brightness and viewing angles, and you’ll want to think about power — a USB lead to a wall box is the usual answer. And the panels need to live on your Wi-Fi well, since every interaction round-trips through Home Assistant.

Still, the value proposition is hard to beat: room touch controls for less than the cost of a takeaway dinner, managed from the same UI as everything else in the house. If you’ve been priced out of the wall-tablet dream, this is the project to try this weekend — and a good reminder that the sharpest edge of home automation isn’t new protocols, it’s someone removing the last excuse not to build the thing.

Posted on

The portable network toolkit: four tools that live on my USB stick

There’s a corner of the toolbox that never seems to make it to the cloud: the USB stick you grab when a network is misbehaving. An XDA writer’s recent kit list is a good excuse to build one of your own — four small, mostly-portable tools that together cover most day-to-day troubleshooting on a home or small-office network.

Wireshark Portable is the heavyweight of the four. It captures and dissects packets as they fly, which sounds arcane until the day you need it: the smart TV that phones home constantly, the app that saturates the uplink, the device talking to something it shouldn’t be. The portable build runs straight off the stick with no install, which matters when you’re working on a machine you don’t administer.

Nmap answers the question every network owner eventually asks: “what is actually on my network?” A quick scan lists every live device, its IP, and the ports it has open. It’s how you find the forgotten IoT plug, confirm the new server is reachable, or spot a service listening that shouldn’t be. Its bundled netcat can even shuffle a file between two machines when nothing else will. Nmap runs on everything — Windows, macOS, Linux — and it belongs on the stick in whatever flavour matches the machines you touch.

DNS Jumper is the niche one, and Windows-only, but handy for a specific job: flipping a machine between DNS servers (Cloudflare, Google, Quad9, your own) to test whether “the internet is slow” is really “DNS is slow”. On Linux the same experiment is a resolvectl command, but for family and client Windows machines, one click beats editing adapter settings by hand.

NetResView is an old NirSoft utility that still earns its slot: it enumerates everything visible on the local network — shares, printers, other machines — in one flat list. It’s the fastest way to answer “why can’t this PC see the printer” without walking the Network neighbourhood of every desktop involved. Being ancient, run it from the stick and expect the odd quirk on modern Windows; for a ten-second inventory it’s still worth it.

The bigger lesson is the USB stick itself. Everyone has a drawer of orphaned 8GB drives, and a labelled toolkit stick is one of the best second lives they can have: portable apps, a couple of bootable installers, config templates, and the network kit above. When something breaks at 9pm on a Sunday, “plug in the stick” beats “download six installers on the machine that’s misbehaving” every time — and unlike cloud tools, it works when the network is the problem.

Posted on

A Raspberry Pi Zero makes a surprisingly good home lab subnet router

Every home lab eventually collides with the same wall: you want to reach your machines from outside, and your ISP won’t let you. Carrier-grade NAT — increasingly common on consumer and NBN connections — means there’s no public IPv4 address and no port forwarding, so the classic “expose a service on the router” playbook simply doesn’t exist. A writer at XDA Developers recently documented their solution, and it’s a lovely piece of lateral thinking: they gave the oldest, weakest board in the drawer — an original Raspberry Pi Zero — the most important job in the lab.

The ingredient that makes it work is a mesh VPN such as Tailscale. Instead of opening ports, every device you care about joins a private overlay network and talks to its peers over encrypted WireGuard tunnels, using UDP hole-punching to sneak through NAT. No inbound ports, no cloud server rental, and the free tier is generous enough for home and small-office use. It’s the answer to CGNAT that doesn’t involve calling your ISP or paying for a static IP you can’t get.

The clever part is how little of it you actually need to install. Installing the VPN client on every server, VM and container gets old fast — especially if you rebuild nodes regularly for experiments. The fix is a subnet router: one device joins the mesh and advertises your local subnet as a route. From that moment, anything connected to the VPN can reach everything on your LAN — servers, printers, smart gadgets, the lot — with no client installed anywhere else. One board, one configuration, whole-network coverage.

And that’s where the Pi Zero earns its keep. Routing packets between a VPN tunnel and your LAN is almost no work at all for a modern CPU, let alone a humble ARM11. The XDA author paired theirs with DietPi to strip the OS down to essentials, enabled IP forwarding, and enabled route advertisement with a single command:

tailscale set --advertise-routes=192.168.0.0/24

Approve the route once in the admin console and you’re done. That $15 board everyone wrote off is now the front door to your entire network.

A few practical notes before you raid the parts drawer:

  • Connect it wired if you can. The original Zero has no Ethernet port, and subnet routing over Wi-Fi is fine for SSH and dashboards but will bottleneck big file transfers. A USB Ethernet adapter helps; a Pi Zero 2 W (or any spare Pi) is better again if you have one.
  • Keep a fallback path. A subnet router is a single point of failure. If remote access matters to you, a second device advertising the same route elsewhere on the LAN gives you failover — or at least a way in when you unplug the wrong cable.
  • Lock the tailnet down. Turn on device approval, use Tailscale’s ACLs to limit which devices can reach which subnets, and consider an exit node so roaming laptops can keep lab access off public Wi-Fi.
  • Mind the power supply. A flaky USB charger will brown out a Pi under network load and you’ll spend an evening debugging a problem that costs $10 to fix.

For small businesses the same pattern applies with a straight face: secure remote access to an office LAN without opening a single inbound port, without a VPN concentrator appliance, and with the audit trail of who connected sitting in one admin console. Not bad for the board at the bottom of the drawer — and a good reminder that in networking, the boring jobs are often the important ones.

Posted on

Cutting AI model costs with OpenAI-compatible API gateways

AI APIs have quietly become a line item for a lot of small businesses. A customer-service assistant here, a document summariser there, and suddenly the monthly bill is measured in hundreds of dollars — most of it spent on frontier models doing work a cheaper model could handle. The good news is that this is one of the few areas in tech where switching costs are close to zero, and a new wave of services has grown up around exactly that.

The pattern looks like this: instead of holding separate accounts with OpenAI, Anthropic, Google and the rest, you point your code at a single gateway that speaks the OpenAI API dialect — the de facto industry standard. Behind that one endpoint, the service routes your requests to whichever provider and model you ask for, at prices that are often well below list. One recently launched example, CheaperInference, promises access to leading models through one OpenAI-compatible API with savings of up to 60% on inference costs. It is new enough that there is little independent coverage yet, so treat it as an example of the category rather than an endorsement — but the category itself is well established.

Why does this work? Because the OpenAI API format became the lingua franca of the industry. Nearly every model provider and open-source framework speaks it, which means an aggregator only needs to translate at the edges. For you, that means the migration is a one-line change — swap the base URL and API key — and your existing code, SDKs and tooling keep working. That same property is your negotiating leverage: when a better price appears elsewhere, you can act on it in minutes, not weeks.

The real savings, though, come from pairing a gateway with some straightforward habits:

  • Route by task, not by default. Most workloads don’t need a frontier model. Classification, extraction, drafting and routing tasks run happily on smaller models that cost a tenth as much. Keep the expensive models for the genuinely hard requests.
  • Measure before and after. Run the same real-world prompts across two or three models and compare quality against cost. You will usually find one or two tasks where the premium model earns its price, and several where it does not.
  • Cache and batch. Identical requests don’t need to hit the API twice, and most providers offer half-price batch processing for work that isn’t time-critical.
  • Watch the context, not just the price. Long documents fed to a “cheap” model with a large context window can cost more than a smart model given a summary. Trim what you send.

There are caveats worth respecting. Your prompts and data pass through a third party, so check the privacy terms before sending anything sensitive — for regulated data, this may rule smaller aggregators out entirely. Latency can differ from going direct. And a very new service deserves a cautious rollout: start with internal, low-stakes workloads, keep an eye on uptime, and keep that one-line switch back to the direct provider ready.

For homelab tinkerers and small businesses alike, this is a rare win-win. The competitive market and the common API dialect mean the cost of experimentation is minutes of your time, and the payoff is a bill that scales with what your workload actually needs rather than what the default model charges. Services like CheaperInference are worth a look — the worst case is you learn where your money was going.

Posted on

WordPress and Static websites on S3

WordPress and Static websites on S3 are two very distinct hosting solutions. WordPress is a Content Management System (CMS) that allows users to easily build and maintain dynamic, content-rich websites. On the other hand, Static websites on S3 are a more traditional hosting solution where HTML, CSS, and JavaScript are all hosted on Amazon’s Simple Storage Service (S3). Both solutions offer different benefits depending on the type of website you are building, so it is important to weigh the pros and cons of each before making a decision.

This website is hosted on Amazon Web Services’ Simple Storage Service (S3). The content management system (CMS) used to edit the website is WordPress.

This is achieved by running a WordPress installation on a local server. The website is then edited as any other WordPress website. A static site exporter is then used to convert the WordPress website into a set of static HTML, CSS, and images. Finally, Rclone is used to sync these files with an S3 bucket.

Once the static files have been synced to S3, they can be hosted as any other static website. This cycle of editing, exporting, and syncing can be repeated as often as necessary.

This method does have its limitations. Any pages that require dynamic content, such as an online store, will not be supported by a static export. However, for websites that can work within these constraints, hosting costs are greatly reduced and the website can be hosted on Amazon Web Services’ Content Delivery Network (CDN) for lightning-fast speeds from anywhere in the world.

Posted on

HITS Stores

I recently launched two online stores that are built on WordPress/Woocommerce platforms. The first website, 800adventures.com.au, is targeted towards CF Moto 800MT owners. This website is hosted on AWS Lightsail and uses Lightsail’s Proxy CDN to cache images and static files with recommended settings for WordPress dynamic websites. It achieves a score of A on GTMetrix.

The second website, getpunnyshirts.com, offers both self-designed and reseller T-shirts. While this website currently has lower sales than 800 Adventures, it is hosted on my home network due to its lower traffic volume. To improve its performance, I am using a proxy on Lightsail with a public IP, which runs a dockerised installation of Nginx configured as a forward proxy with SSL certificates for the domain. The traffic is forwarded via a Zerotier VPN PtP network to my home Proxmox cluster that hosts the web server. Unfortunately, due to the time it takes to receive the first byte, the website only achieves an F on GTMetrix. However, the bandwidth is sufficient for low traffic.

WordPress Appliance - Powered by TurnKey Linux