I survived the Battle against the Army of Dead

I saw them, the mindless zombies, raging towards the ones I care about, aiming to loot the resource I myself and the ones I love an care about preciously gathered with sweat and blood. They're sent by the AI overlords, to gather to take down all the resources to themselves. They do not sleep they do not stop, they do not care how deep they suck the soul out. I could have easily just hid behind cloudflare and serve all my traffic in peace. But nah I chose to take things to my hand. It's not the first time I'm facing these icy monsters, I've seen them drilling onto my site and taking it down before also. but this time I was prepared. It all started on July 1st. I observed reccuring crashes to my mediawiki pods, they kept failing again and again and again. I was under the impresson they were failin' because of the hacks i put up to solve the many compatibility issues., but soon realised that's not the case. the main one being the one called `meta-webindexer` riding in on real, headless Chrome. It didn't knock. It solved our door's puzzle instantly and let itself in, then began hammering Special:RecentChangesLinked — a page that, under the hood, runs a full table scan every single time someone asks for it.

Each of those queries took five to fifteen seconds. It didn't matter. The bot didn't ask once. It asked hundreds of times, in parallel, across every parameter permutation it could construct.

Twenty PHP workers stood between that traffic and the database. The math was never going to work in our favor: twenty workers, eight seconds a query, means the entire pool saturates at just two and a half requests per second. This thing was sending far more than that.

Thirty-three restarts. Thirty-three hours. Every single time, the pod died quietly — exit code 0, no scream, no crash report, just supervisord shrugging and kubelet pulling the trigger after three failed health checks. From the outside it looked like routine restarts. It was a wall going down, over and over, and getting rebuilt just in time to fall again.

Just when the first wave looked contained, a second force showed up — and this one wasn't a single overzealous bot. It was 282 unique IP addresses in a single ten-minute window. A botnet, rotating through eight different browser identities, each one dressed convincingly as an ordinary human on Chrome or Edge, each one carrying a Referer header that pretended it had arrived by clicking a normal link.

They didn't come through the front gate either. They went straight for Special:Drilldown — a page that runs expensive multi-table joins with arbitrary filters, each one costing five to ten seconds of database time. Every request looked almost human. Every request was designed to stay just under the per-IP limits we'd set, by simply not repeating the same IP twice.

This is the moment every siege story turns: when you realize the enemy isn't one large creature you can strike down, but a horde, individually unremarkable, collectively unstoppable by any single wall.

I had setup an anubis challenge but haven't really cared to really polish it to sharpness and it wasn't doing any harm to any of the bots. I had to dial it up, but this was hurting my actual users instead.

Rebuilding the Wall, Properly this Time.

Rebuilding the Wall — Properly, This Time

So we stopped trying to build one big gate and built five smaller ones instead.

The first line (CrowdSec): this is the piece that actually turned the tide, and it stands ahead of everything else. A firewall bouncer runs directly on the host, outside Kubernetes entirely, enforcing a community-sourced blocklist of known-bad IPs at the kernel level before Traefik, before Anubis, before a single service even has to think about the request. Seventeen thousand-plus IPs get dropped by iptables before they touch anything we run. This layer takes the bulk of the botload and doesn't even break a sweat doing it. When we briefly turned it off to check something, Traefik's CPU load jumped by over 130% in minutes as all seventeen thousand previously-banned IPs came flooding straight back in. Turning it back on dropped it instantly. That's not a marginal improvement. That's the difference between a wall with sentries and a wall without them.

The second line (Traefik): whatever wasn't already known-bad and makes it past the kernel hits a hard rate and concurrency ceiling — three simultaneous requests per IP on anything under /Special:, one request per second sustained, instant 503 past that, at the load balancer, before a single database connection opens.

The third line (Anubis): the handful that survive both of those get handed a proof-of-work puzzle dialed up to difficulty 10 — a trillion hashes, effectively unsolvable in any reasonable time even on a GPU. Anything short of dedicated, purpose-built hardware just sits there grinding forever. We also closed the one gap that mattered: the original rule only checked pretty URLs like /Special:Drilldown/..., and the second wave simply asked for the same page through a query string instead, walking straight past the puzzle. A CEL expression now inspects the decoded query parameter too. There is no side door anymore.

The last punch (MediaWiki itself): for anything that somehow survives all of the above, there's one final, blunt, unglamorous check: if you're not logged in, you don't get to execute a Special: page. Full stop. The permission check happens before any query touches the database — no cleverness, just a flat refusal at the door.

The garrison behind the walls (PHP-FPM): not a gate at all, just insurance for whatever still gets through — we doubled the worker pool and cut the timeout on stuck workers from five minutes to ninety seconds, so even a single slow query can no longer hold the whole line closed behind it.

The War Isn't Over

Here's the honest ending, because the honest ending is more interesting than a clean one: the botnet never left. As I write this, all 282-plus IPs are still out there, still probing, still trying. The difference is that now, every single one of those requests dies at one of three layers before it can cost us anything dropped at the kernel by the firewall, denied at the edge by Anubis, or refused outright by MediaWiki itself. Zero database queries. Zero PHP workers spent. Zero restarts.

That's not victory in the sense of the army marching home. It's victory in the Jon Snow sense — the wall held, the night is still long, and you don't get to stop watching just because you won this fight. You just get to go back up the stairs and keep watching, a little wiser about where the ice was thin.

If there's a lesson here for anyone else running open infrastructure in the age of AI-scale scraping, it's this: stop looking for the one silver sword. There isn't one. Build boring, layered, unglamorous defenses — a deny list here, a concurrency cap there, a firewall bouncer running quietly on the host — and let each one do exactly one job. The Wall was never one wall. It was ice, and men, and ravens, and everyone doing their part.

wiki.fosscell.org is up. The night watch continues.