casino/docs/SCALING.md

# Scaling by cloning

The app is stateless. To serve more players, clone the app VM and boot it.
An instance works out what it is on startup: it generates its own identity,
registers itself, and negotiates which games it drives. Nothing is assigned by
hand, and no file needs editing after a clone.

## What runs where

| VM | Runs | How many |
|---|---|---|
| **core** | PostgreSQL + Redis + Caddy | exactly one |
| **app** | `quantum-arcade` | **clone this one** |
| **lightning** | Alby Hub | one, firewalled |

**Do not clone the core VM.** If each app clone brings its own PostgreSQL and
Redis, the clones share nothing: separate ledgers, separate rounds, mutually
invisible. The app VM must contain *only* the arcade binary.

Keep Alby Hub separate from the app. The app VMs are what every phone talks to;
the Lightning node holds keys and channel state. Separation is what makes a
compromised app instance survivable — it holds a budget-capped credential, not
the node.

## Configuring a clone

Two variables, both pointing at the core VM:

```bash
ARCADE_DSN=postgres://arcade:PASSWORD@10.0.0.10:5432/arcade
ARCADE_REDIS=10.0.0.10:6379
```

Optionally, if the instance's routable address cannot be detected (multiple
NICs, NAT):

```bash
ARCADE_ADVERTISE=10.0.0.21:8080
```

Otherwise it advertises the first non-loopback IPv4 address it finds, which is
correct on a normal Proxmox bridge with DHCP.

Everything else — instance id, which games it drives, which peers exist — is
determined at runtime.

## How instances divide the work

Each game is driven by exactly one instance at a time.

- On startup an instance **campaigns** for each game: a Redis key set with
  `SET NX PX`, held for `LeaseTTL` (6s) and renewed every 2s.
- The winner runs that game's round loop, settles to the ledger, and publishes
  every frame to Redis.
- Every other instance **relays** those frames to its own connected clients.
  A client cannot tell which instance it is attached to.
- Bets and cash-outs arriving at a non-leader are **forwarded** to the leader,
  because only the leader holds the authoritative round state. Sessions live in
  Redis, so a token issued anywhere is accepted everywhere and the forwarded
  request authenticates normally.

Leadership spreads itself across instances naturally: whichever instance
campaigns first for a given game gets it, so three games across two instances
lands roughly 2/1.

## Failure

An instance dying is not a special case. Its lease stops being renewed, expires
within `LeaseTTL`, and the next campaign hands its games to a survivor.

Measured with a hard `kill -9` on an instance leading two of three games:

```
t+0s   killed
t+6s   both games taken over, rounds running
```

Six seconds, unattended. Players attached to the dead instance reconnect
through the load balancer and rejoin whichever instance answers.

The in-flight round on the dead instance produces no outcome. Because stakes
are debited when a bet is placed, those players would otherwise be quietly
short — the books stay balanced, but the money sits with the house.

Every instance therefore runs a reconciler every 30 seconds. It finds rounds
left unresolved past a staleness window, marks them **void** (not settled: an
abandoned round has no outcome, so there is no seed to reveal), and refunds
every stake. Claiming the round happens before any money moves, so concurrent
reconcilers on different instances refund exactly once.

## Load balancing

Caddy needs no sticky sessions — any instance serves any request.

```
arcade.lan {
    reverse_proxy 10.0.0.21:8080 10.0.0.22:8080 10.0.0.23:8080 {
        lb_policy least_conn
        health_uri /api/health
        health_interval 5s
    }
}
```

`least_conn` suits long-lived WebSockets better than round-robin, which
distributes connection *attempts* rather than connections.

## Watching the fleet

```bash
curl -s http://arcade.lan/api/cluster | jq
```

Returns every registered instance, which one drives each game, and which
instance answered. Useful for confirming a clone joined, and for watching
leadership move during a failover.

## Measured capacity

Run against one instance on a 4-core / 7GB box, with the load generator on the
*same machine* competing for CPU — so these are conservative:

| Connections | Failed | Dial p50 / p99 | Server RSS |
|---|---|---|---|
| 500 | 0 | 1ms / 11ms | — |
| 3,000 | 0 | 1ms / 122ms | — |
| 10,000 | 0 | 1ms / 15ms | 258 MB |
| 25,000 | 0 | 2ms / 1.33s | 586 MB |

About **26KB of server memory per connection**, so 50,000 connections is
roughly 1.2GB — comfortable on any real machine. Connection capacity is not
the constraint people expect it to be.

Reproduce with:

```bash
go run ./cmd/loadtest -conns 10000 -duration 30s -ramp 20s
```

The dial p99 at 25k reflects both processes sharing four cores; on separate
machines it is far lower. Rising dial latency is the signal to add an instance.

## Where this stops scaling

Adding app clones raises the ceiling on connections and fan-out. It does not
raise these:

- **Bet throughput**, measured at ~230/sec, is bounded by PostgreSQL commit
  cost. Every clone contends for the same database. Getting past this needs
  in-memory balance reservation with batched persistence — a change to how
  money is held, not a deployment change.

  This is the real ceiling, and it is worth being precise about what it means:
  50,000 people can *watch* comfortably, and tens of thousands can hold
  connections on a single instance. What they cannot all do is place a bet in
  the same twenty-second window. A 20s window absorbs roughly 4,600 bets.
- **A single game's round loop** runs on one instance, by design. A game cannot
  be split across instances without a distributed clock.

So: clone freely for more spectators and more connections. For more *bets per
second*, the database is the thing to work on.