feat(cluster): zero-config horizontal scaling by cloning

An instance decides what it is at startup instead of being told: it
generates its own identity, registers a heartbeat, and campaigns for
each game. Exactly one instance drives a game's rounds and publishes
frames; the rest relay them and forward mutations to the leader. Clone
the VM, boot it, done.

Sessions and the scratch nonce move to Redis. Both were per-instance
state that would have broken behind a load balancer: a token minted by
one clone was unknown to the others, and two clones would have handed
the same nonce to different players, which for the same key means the
same outcome.

Fixes a bug found by running two instances: /api/games read the local
room object, so a follower reported a permanently settled game and its
clients never saw a betting window. Hubs now serve the last frame they
saw, produced or relayed.

Failover measured at 6s after kill -9 on an instance leading two games.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
drjones
2026-08-05 23:07:47 +00:00
parent 038550b6ff
commit c12640cf52
10 changed files with 1306 additions and 40 deletions

View File

@@ -123,6 +123,20 @@ bugs: a posting set that minted 18 quintillion millisatoshis by wrapping the
zero-sum check, a crash point that overflowed negative at the rarest seed, and
scratch tables that advertised 98% while paying 56%.
## Scaling
The app is stateless: clone the VM and boot it. An instance generates its own
identity, finds its peers through Redis, and campaigns for the games it will
drive. Exactly one instance runs a given game's rounds; the rest relay its
frames and forward bets to it.
An instance dying is not a special case — its lease expires and a survivor
takes over. Measured at six seconds, unattended, after a `kill -9`.
Clone the **app** VM only. PostgreSQL and Redis stay on one shared machine;
cloning those gives every instance its own ledger and they share nothing.
Full topology in [docs/SCALING.md](docs/SCALING.md).
## Status
Built and tested: