We Took Ourselves Offline on Purpose. Here's What We Learned.

Last Friday, we told you we were going to break things.

We did.

This past weekend, Anton Payments ran a full disaster recovery exercise — deliberately taking down our website and portals to see exactly how our systems behave when connectivity disappears and the pressure is on. Not in a staging sandbox. On the real thing. On our terms, on a schedule we chose, with the whole team watching.

We promised we'd share what we learned. So here it is.

Why a pre-launch company would pull its own plug

It's a fair question. We haven't even launched. Why go looking for trouble?

Because in cross-border payments, reliability isn't a feature you bolt on later — it's the product. When you're moving money across borders, rails, and jurisdictions, "mostly up" isn't a standard anyone should accept. The moment a payout stalls, you're not debugging a website. You're standing between someone and money they're counting on.

We'd rather find our gaps now, quietly, on our own terms — than discover them later, loudly, on yours.

So we treated this the way we treat everything at Anton: assume it will fail, then prove it won't. And when the day comes that something does fail, we want the recovery to be muscle memory, not improvisation.

What we actually did

We simulated a real connectivity loss and took our public website and client portals offline. Then we watched two things at once:

Automated recovery — the systems designed to detect failure and restore service without a human touching anything. Health checks, failover, the whole automated response layer.

Manual runbooks — the step-by-step procedures a human follows when the automation isn't enough. Because the ugly truth of disaster recovery is that automation handles the failures you predicted. Runbooks are for the ones you didn't.

The goal wasn't just "does it come back up." It was: how fast, how cleanly, and where does the plan quietly fall apart under real conditions.

How it went

Short version: it worked — and it taught us things.

We ran the platform through a series of escalating failure scenarios, from a single service dropping to a full system-down catastrophe, and measured recovery at every level:

Single service failure: ~3 minutes from TTFF (Time To First Fail) to fully automated restoration. No human intervention required.
Database corruption: 6 minutes from failure to a fully automated restore from a just-in-time backup.
Complete system failure (manual recovery): 78 minutes from a Tier I support notification to full resolution — a total catastrophic failure, brought back entirely by hand following our runbooks.
Failover from primary to disaster-recovery cluster: 255 seconds — roughly 4 minutes — for a fully automated DNS reroute to the DR cluster.

That 78-minute number is the one we want to sit with for a second. That was the worst-case scenario — everything down, no automation to lean on, a human working from a runbook under pressure. Recovering a complete system failure by hand in under an hour and a half isn't luck. It's the difference between a runbook that exists and a runbook that works.

And here's the part we're actually proudest of: we found gaps.

Gap one — notification latency. Our manual incident notification is supposed to fire within a 90-second SLA. It ran about 4 minutes over, and the culprit was SMS carrier delay — a single-channel dependency we didn't want to have. The fix is already in motion: we're forcing multi-channel notifications across app, SMS, Slack, and email, so no single carrier's lag can ever hold up an alert again.

Gap two — auth persistence during recovery. Partway through recovery, hardware-secured keys on Google locked for reasons that weren't obvious at first. Root cause: a local agent's authentication wasn't persisting through the recovery process. The fix is a standardized session-start script that every support agent runs at the top of an incident — logging every action taken, giving us a reproducible record, and surfacing state and auth issues before they can bite.

That's not a failure of the test. That's the entire point of the test. Every one of those findings is a problem we now solve in calm daylight instead of at 3 a.m. during a real incident with real money in flight. Both are logged, both have owners, and both are already being fixed.

Why this matters long-term

A disaster recovery exercise isn't a box you check once and frame on the wall. It's a muscle. And muscles atrophy if you don't work them.

This is the first of what will be a regular practice at Anton — deliberately testing our own resilience, on a cadence, forever. Because the platforms our future clients will trust with cross-border payouts don't earn that trust with a nice landing page. They earn it by proving, over and over, that when things go wrong, the money still moves and the lights come back on.

Anton Payments isn't "using AI." Anton Payments is the AI. But the intelligence layer that makes our payout decisions is only as valuable as the infrastructure underneath it is unshakeable. Smart decisions on a fragile foundation are still fragile. So we're building both — the judgment and the backbone — from day one.

We came out of this weekend with a stronger, battle-tested platform, a sharper runbook, and a team that's now practiced the hard scenario instead of just talking about it.

We told you we'd break things and share what we learned.

Consider it shared.