casino/docs/SCALING.md

Scaling by cloning

The app is stateless. To serve more players, clone the app VM and boot it. An instance works out what it is on startup: it generates its own identity, registers itself, and negotiates which games it drives. Nothing is assigned by hand, and no file needs editing after a clone.

What runs where

VM	Runs	How many
core	PostgreSQL + Redis + Caddy	exactly one
app	quantum-arcade	clone this one
lightning	Alby Hub	one, firewalled

Do not clone the core VM. If each app clone brings its own PostgreSQL and Redis, the clones share nothing: separate ledgers, separate rounds, mutually invisible. The app VM must contain only the arcade binary.

Keep Alby Hub separate from the app. The app VMs are what every phone talks to; the Lightning node holds keys and channel state. Separation is what makes a compromised app instance survivable — it holds a budget-capped credential, not the node.

Configuring a clone

Two variables, both pointing at the core VM:

ARCADE_DSN=postgres://arcade:PASSWORD@10.0.0.10:5432/arcade
ARCADE_REDIS=10.0.0.10:6379

Optionally, if the instance's routable address cannot be detected (multiple NICs, NAT):

ARCADE_ADVERTISE=10.0.0.21:8080

Otherwise it advertises the first non-loopback IPv4 address it finds, which is correct on a normal Proxmox bridge with DHCP.

Everything else — instance id, which games it drives, which peers exist — is determined at runtime.

How instances divide the work

Each game is driven by exactly one instance at a time.

• On startup an instance campaigns for each game: a Redis key set with SET NX PX, held for LeaseTTL (6s) and renewed every 2s.
• The winner runs that game's round loop, settles to the ledger, and publishes every frame to Redis.
• Every other instance relays those frames to its own connected clients. A client cannot tell which instance it is attached to.
• Bets and cash-outs arriving at a non-leader are forwarded to the leader, because only the leader holds the authoritative round state. Sessions live in Redis, so a token issued anywhere is accepted everywhere and the forwarded request authenticates normally.

Leadership spreads itself across instances naturally: whichever instance campaigns first for a given game gets it, so three games across two instances lands roughly 2/1.

Failure

An instance dying is not a special case. Its lease stops being renewed, expires within LeaseTTL, and the next campaign hands its games to a survivor.

Measured with a hard kill -9 on an instance leading two of three games:

t+0s   killed
t+6s   both games taken over, rounds running

Six seconds, unattended. Players attached to the dead instance reconnect through the load balancer and rejoin whichever instance answers.

The in-flight round on the dead instance is lost — bets already written to the ledger stand, and the round simply never settles. This is the one rough edge: stakes are debited at bet time, so a round lost mid-flight leaves those stakes with the house. A reconciliation job that refunds unsettled rounds is not yet built.

Load balancing

Caddy needs no sticky sessions — any instance serves any request.

arcade.lan {
    reverse_proxy 10.0.0.21:8080 10.0.0.22:8080 10.0.0.23:8080 {
        lb_policy least_conn
        health_uri /api/health
        health_interval 5s
    }
}

least_conn suits long-lived WebSockets better than round-robin, which distributes connection attempts rather than connections.

Watching the fleet

curl -s http://arcade.lan/api/cluster | jq

Returns every registered instance, which one drives each game, and which instance answered. Useful for confirming a clone joined, and for watching leadership move during a failover.

Where this stops scaling

Adding app clones raises the ceiling on connections and fan-out. It does not raise these:

• Bet throughput, measured at ~230/sec, is bounded by PostgreSQL commit cost. Every clone contends for the same database. Getting past this needs in-memory balance reservation with batched persistence — a change to how money is held, not a deployment change.
• A single game's round loop runs on one instance, by design. A game cannot be split across instances without a distributed clock.

So: clone freely for more spectators and more connections. For more bets per second, the database is the thing to work on.