# Scaling by cloning

The app is stateless. To serve more players, clone the app VM and boot it.
An instance works out what it is on startup: it generates its own identity,
registers itself, and negotiates which games it drives. Nothing is assigned by
hand, and no file needs editing after a clone.

## What runs where

| VM | Runs | How many |
|---|---|---|
| **core** | PostgreSQL + Redis + Caddy | exactly one |
| **app** | `quantum-arcade` | **clone this one** |
| **lightning** | Alby Hub | one, firewalled |

**Do not clone the core VM.** If each app clone brings its own PostgreSQL and
Redis, the clones share nothing: separate ledgers, separate rounds, mutually
invisible. The app VM must contain *only* the arcade binary.

Keep Alby Hub separate from the app. The app VMs are what every phone talks to;
the Lightning node holds keys and channel state. Separation is what makes a
compromised app instance survivable — it holds a budget-capped credential, not
the node.

## Configuring a clone

Two variables, both pointing at the core VM:

```bash
ARCADE_DSN=postgres://arcade:PASSWORD@10.0.0.10:5432/arcade
ARCADE_REDIS=10.0.0.10:6379
```

Optionally, if the instance's routable address cannot be detected (multiple
NICs, NAT):

```bash
ARCADE_ADVERTISE=10.0.0.21:8080
```

Otherwise it advertises the first non-loopback IPv4 address it finds, which is
correct on a normal Proxmox bridge with DHCP.

Everything else — instance id, which games it drives, which peers exist — is
determined at runtime.

## How instances divide the work

Each game is driven by exactly one instance at a time.

- On startup an instance **campaigns** for each game: a Redis key set with
  `SET NX PX`, held for `LeaseTTL` (6s) and renewed every 2s.
- The winner runs that game's round loop, settles to the ledger, and publishes
  every frame to Redis.
- Every other instance **relays** those frames to its own connected clients.
  A client cannot tell which instance it is attached to.
- Bets and cash-outs arriving at a non-leader are **forwarded** to the leader,
  because only the leader holds the authoritative round state. Sessions live in
  Redis, so a token issued anywhere is accepted everywhere and the forwarded
  request authenticates normally.

Leadership spreads itself across instances naturally: whichever instance
campaigns first for a given game gets it, so three games across two instances
lands roughly 2/1.

## Failure

An instance dying is not a special case. Its lease stops being renewed, expires
within `LeaseTTL`, and the next campaign hands its games to a survivor.

Measured with a hard `kill -9` on an instance leading two of three games:

```
t+0s   killed
t+6s   both games taken over, rounds running
```

Six seconds, unattended. Players attached to the dead instance reconnect
through the load balancer and rejoin whichever instance answers.

The in-flight round on the dead instance is lost — bets already written to the
ledger stand, and the round simply never settles. This is the one rough edge:
stakes are debited at bet time, so a round lost mid-flight leaves those stakes
with the house. A reconciliation job that refunds unsettled rounds is not yet
built.

## Load balancing

Caddy needs no sticky sessions — any instance serves any request.

```
arcade.lan {
    reverse_proxy 10.0.0.21:8080 10.0.0.22:8080 10.0.0.23:8080 {
        lb_policy least_conn
        health_uri /api/health
        health_interval 5s
    }
}
```

`least_conn` suits long-lived WebSockets better than round-robin, which
distributes connection *attempts* rather than connections.

## Watching the fleet

```bash
curl -s http://arcade.lan/api/cluster | jq
```

Returns every registered instance, which one drives each game, and which
instance answered. Useful for confirming a clone joined, and for watching
leadership move during a failover.

## Where this stops scaling

Adding app clones raises the ceiling on connections and fan-out. It does not
raise these:

- **Bet throughput**, measured at ~230/sec, is bounded by PostgreSQL commit
  cost. Every clone contends for the same database. Getting past this needs
  in-memory balance reservation with batched persistence — a change to how
  money is held, not a deployment change.
- **A single game's round loop** runs on one instance, by design. A game cannot
  be split across instances without a distributed clock.

So: clone freely for more spectators and more connections. For more *bets per
second*, the database is the thing to work on.