Our health check was green for fifty days while every download failed

The dashboard was green. Everything was broken.

We ran a routine audit of our own stack on 24 July. Health endpoint: 200 OK. Redis: 1 ms round trip. Containers healthy, seven weeks of uptime, zero restarts, 14% disk. Load average 0.36 on six cores.

Then we pasted an Instagram Reel into our own tool and got an error page.

So we pasted another one. Same error. Every content type, every URL shape, same result — and the traffic logs showed 200 to 400 people a day had been hitting that same wall for roughly seven weeks. Nobody emailed. They just left.

That last part is worth sitting with. A free tool has no support ticket queue to warn you. Users don't file bugs; they close the tab and open a competitor. Our only signal that the product worked was that we hadn't personally checked.

What our health check was actually checking

Here's the endpoint, more or less verbatim as it existed that morning:

It pinged Redis. That's the whole check. If Redis answered PONG, the service reported healthy, Docker's healthcheck stayed green, and any uptime monitor pointed at it saw 200 forever.

Redis was never the fragile part. The fragile parts are a residential proxy pool leased from a third party, a binary that fights an adversarial API, and Instagram itself — none of which the check touched. We had instrumented the one dependency that never breaks.

This is the failure mode worth naming, because it's not laziness — the Redis ping was easy to write and easy to reason about, which is exactly why it got written. A health check that measures only what's convenient to measure doesn't just miss outages. It manufactures confidence while it misses them. Green for fifty days is worse than no dashboard at all, because no dashboard would have made us check manually.

Root cause one: the pool had been 407ing since June

Every Instagram fetch we make exits through a residential proxy — datacenter IPs get a 403 from Instagram's CDN before the response headers finish parsing, which we wrote up separately in why datacenter IPs can't fetch Instagram.

Testing the pool from the box took one loop. All 100 proxies returned 407 Proxy Authentication Required. Not a subset. Not intermittent. Every single one, instantly.

A 407 across an entire pool is never a network problem — it's an account problem. Expired subscription, rotated password, exhausted bandwidth quota. Ours had lapsed and we hadn't noticed, because nothing was watching the thing that mattered.

The last container log line proving the proxies still authenticated was dated 2 June. Successful parses don't log, so the exact death date is unrecoverable. Somewhere between 2 June and 24 July, the credentials stopped working and the product stopped existing.

Root cause two: yt-dlp had been frozen since March

New credentials went in. The proxy probe went green. Parses still failed — with a different error this time:

ERROR: [Instagram] DbB4is3D0GM: Instagram sent an empty media response.

The obvious suspicion was that the URL had rotted, so we pulled two fresh public Reels straight off @nasa's profile page and tried those. Same empty response. Not the URL.

Then we checked the version in the container. 2026.03.17. Upstream at that moment: 2026.07.04. Four months behind, on a tool whose entire job is tracking an API that actively fights it.

The Dockerfile said apk add yt-dlp. That one line is the whole bug. Distro packages lag by design — Alpine's maintainers batch and stabilise, which is correct behaviour for a C library and catastrophic for a scraper. Instagram ships anti-scrape changes every few weeks; yt-dlp patches within days; Alpine picks it up whenever. Pinning the upstream binary and bumping it deliberately is the only sane pattern here, and yt-dlp's own docs have said so for years.

A twenty-minute detour into zero padding

Switching to the upstream binary should be four lines of Dockerfile. It cost us a wrong turn worth documenting.

PyPI reports the version as 2026.7.4. We built the release URL from that string and got a 404. The tag on GitHub is 2026.07.04 — zero-padded month and day.

Two representations of the same release, one of them the only one that resolves. If you script a yt-dlp bump, read the tag from the GitHub releases API rather than composing it from a version string, or you'll ship a Dockerfile that fails at build time and looks like a network flake.

We added yt-dlp --version as the last step of that RUN layer. A corrupt or 404'd download now fails the build instead of producing an image that boots fine and returns empty media responses in production.

The new monitor paged us four days later. It was wrong.

The obvious lesson from a fifty-day blind spot is to check the thing that actually breaks, so we shipped a second endpoint that fetches through a randomly chosen proxy and returns 503 when that fetch fails. Deliberately separate from the Docker healthcheck — a dead proxy pool should page a human, not restart a container that's running perfectly well.

It worked. On 28 July at 03:38 UTC it fired, and the alert was wrong.

By the time we looked, all ten proxies answered in 200-600 ms. The site monitor had stayed green through the incident: 363 checks, 363 × 200. No restarts, no OOM, load 0.26, one single parse request in the entire hour.

The nginx access log had the shape of it. 353 probes total, 4 of them 503 — and those four landed at 03:38:20, 03:38:36, 03:38:50, 03:39:05. Fourteen seconds apart, when the configured interval is five minutes.

Caching a failure is not the same as caching a success

Those tight intervals are the uptime monitor's confirmation retries — check fails, re-check quickly a few times, alert only if it keeps failing. Good design. It exists precisely to filter out one-off blips.

Our endpoint cached its verdict in Redis for 60 seconds, to avoid hammering a fixed-IP pool on every poll. Reasonable in isolation. Fatal in combination: the first probe failed, cached degraded, and all three confirmation retries read that cached failure instead of re-probing. Four failed checks, one actual failure. The retry logic was defeated by a cache it couldn't see.

There's a general rule buried in there. Success and failure deserve different TTLs. Caching a success is a load optimisation — the world is fine, don't re-ask. Caching a failure is a decision to keep asserting something is broken without rechecking, and every consumer downstream that tries to verify gets handed your stale opinion.

The second flaw was sample size. One random proxy out of ten, single vote, no quorum. On a residential pool where individual IPs routinely blip for a minute, that's a monitor engineered to cry wolf roughly proportional to pool flakiness.

What we changed

The probe now samples three distinct proxies concurrently and reports degraded only when a majority fail. A single dead IP surfaces as failed: 1 in the payload — visible, not alarming.

Success caches for 60 seconds; failure caches for 15. Confirmation retries now re-probe, which is the entire point of confirmation retries.

Every imperfect probe writes a line to the container log with the sample size, failure count and reason. Before this, the failure reason existed only in an HTTP response body nobody stored — which is why we can tell you that one probe failed on 28 July and cannot tell you why. That gap was ours, and it's closed.

We also stopped rendering raw upstream errors to users. The old failure card printed yt-dlp's stderr verbatim, proxy hostnames and all. Users got implementation details they couldn't act on, and we got a small information leak for free.

If you run something like this

Four things we'd tell a past version of ourselves, in rough order of how much they cost us.

Monitor the dependency that can bill you. Redis doesn't have a subscription that lapses. Your proxy provider, your API vendor and your certificate issuer all do, and none of them will tell your dashboard before they tell your users.

Pin the adversarial dependency and put a reminder on it. Anything in a cat-and-mouse game with a platform — yt-dlp, youtube-dl, an unofficial SDK — should be pinned to an upstream release you bump on a schedule, never pulled from a distro repo. We bump ours roughly monthly now, and treat a sudden drop in parse success as a version check before anything else.

Log outcomes, not just errors. Our old logging recorded exceptions, so a failure that returned a clean structured error object left no trace. Now every parse writes a one-line outcome with type and latency, no user URLs, which turns weekly success rate into a grep instead of a guess.

Ask what a green check actually proves. Ours proved Redis was up. It proved nothing about whether a person could download a video, which was the only question worth asking. Write down the sentence your health check lets you say out loud — if that sentence isn't the product's core promise, the check is decoration.

Fifty days of failure, and the fix was two config lines and a version bump. That ratio is the part we keep thinking about.