casino/docs/SCALING.md

Scaling by cloning

The app is stateless. To serve more players, clone the app VM and boot it. An instance works out what it is on startup: it generates its own identity, registers itself, and negotiates which games it drives. Nothing is assigned by hand, and no file needs editing after a clone.

What runs where

VM	Runs	How many
core	PostgreSQL + Redis + Caddy	exactly one
app	quantum-arcade	clone this one
lightning	Alby Hub	one, firewalled

Do not clone the core VM. If each app clone brings its own PostgreSQL and Redis, the clones share nothing: separate ledgers, separate rounds, mutually invisible. The app VM must contain only the arcade binary.

Keep Alby Hub separate from the app. The app VMs are what every phone talks to; the Lightning node holds keys and channel state. Separation is what makes a compromised app instance survivable — it holds a budget-capped credential, not the node.

Configuring a clone

Two variables, both pointing at the core VM:

ARCADE_DSN=postgres://arcade:PASSWORD@10.0.0.10:5432/arcade
ARCADE_REDIS=10.0.0.10:6379

Optionally, if the instance's routable address cannot be detected (multiple NICs, NAT):

ARCADE_ADVERTISE=10.0.0.21:8080

Otherwise it advertises the first non-loopback IPv4 address it finds, which is correct on a normal Proxmox bridge with DHCP.

Everything else — instance id, which games it drives, which peers exist — is determined at runtime.

How instances divide the work

Each game is driven by exactly one instance at a time.

• On startup an instance campaigns for each game: a Redis key set with SET NX PX, held for LeaseTTL (6s) and renewed every 2s.
• The winner runs that game's round loop, settles to the ledger, and publishes every frame to Redis.
• Every other instance relays those frames to its own connected clients. A client cannot tell which instance it is attached to.
• Bets and cash-outs arriving at a non-leader are forwarded to the leader, because only the leader holds the authoritative round state. Sessions live in Redis, so a token issued anywhere is accepted everywhere and the forwarded request authenticates normally.

Leadership spreads itself across instances naturally: whichever instance campaigns first for a given game gets it, so three games across two instances lands roughly 2/1.

Failure

An instance dying is not a special case. Its lease stops being renewed, expires within LeaseTTL, and the next campaign hands its games to a survivor.

Measured with a hard kill -9 on an instance leading two of three games:

t+0s   killed
t+6s   both games taken over, rounds running

Six seconds, unattended. Players attached to the dead instance reconnect through the load balancer and rejoin whichever instance answers.

The in-flight round on the dead instance produces no outcome. Because stakes are debited when a bet is placed, those players would otherwise be quietly short — the books stay balanced, but the money sits with the house.

Every instance therefore runs a reconciler every 30 seconds. It finds rounds left unresolved past a staleness window, marks them void (not settled: an abandoned round has no outcome, so there is no seed to reveal), and refunds every stake. Claiming the round happens before any money moves, so concurrent reconcilers on different instances refund exactly once.

Load balancing

Caddy needs no sticky sessions — any instance serves any request.

arcade.lan {
    reverse_proxy 10.0.0.21:8080 10.0.0.22:8080 10.0.0.23:8080 {
        lb_policy least_conn
        health_uri /api/health
        health_interval 5s
    }
}

least_conn suits long-lived WebSockets better than round-robin, which distributes connection attempts rather than connections.

Watching the fleet

curl -s http://arcade.lan/api/cluster | jq

Returns every registered instance, which one drives each game, and which instance answered. Useful for confirming a clone joined, and for watching leadership move during a failover.

Measured capacity

Run against one instance on a 4-core / 7GB box, with the load generator on the same machine competing for CPU — so these are conservative:

Connections	Failed	Dial p50 / p99	Server RSS
500	0	1ms / 11ms	—
3,000	0	1ms / 122ms	—
10,000	0	1ms / 15ms	258 MB
25,000	0	2ms / 1.33s	586 MB

About 26KB of server memory per connection, so 50,000 connections is roughly 1.2GB — comfortable on any real machine. Connection capacity is not the constraint people expect it to be.

Reproduce with:

go run ./cmd/loadtest -conns 10000 -duration 30s -ramp 20s

The dial p99 at 25k reflects both processes sharing four cores; on separate machines it is far lower. Rising dial latency is the signal to add an instance.

Where this stops scaling

Adding app clones raises the ceiling on connections and fan-out. It does not raise these:

• Bet throughput, measured at ~230/sec, is bounded by PostgreSQL commit cost. Every clone contends for the same database. Getting past this needs in-memory balance reservation with batched persistence — a change to how money is held, not a deployment change.

  This is the real ceiling, and it is worth being precise about what it means: 50,000 people can watch comfortably, and tens of thousands can hold connections on a single instance. What they cannot all do is place a bet in the same twenty-second window. A 20s window absorbs roughly 4,600 bets.

• A single game's round loop runs on one instance, by design. A game cannot be split across instances without a distributed clock.

So: clone freely for more spectators and more connections. For more bets per second, the database is the thing to work on.