# Operations

Persistence, failover, and tuning for a self-hosted arcade.

## Machines

| Role | Runs | Clone? |
|---|---|---|
| **core** | PostgreSQL, Redis, Caddy | no — one only |
| **app** | `quantum-arcade` | yes, freely |
| **standby** | PostgreSQL + `standby.sh follow` | no — one is enough |
| **lightning** | Alby Hub | no — firewalled |

## Backups

PostgreSQL is the only thing that cannot be rebuilt. The app binary embeds its
own client, and Redis holds only sessions and leases, which regenerate.

```bash
sudo mkdir -p /var/backups/quantum-arcade
./ops/backup.sh init          # once
sudo cp ops/arcade-backup.{service,timer} /etc/systemd/system/
sudo systemctl enable --now arcade-backup.timer
```

Every five minutes it captures a compressed snapshot, keeps a rolling 24 hours
(288 snapshots), and prunes the rest.

**Every dump is checked before it replaces the previous one** — size and format
header. A backup script that reports success on a truncated file is worse than
no backup, because it converts a recoverable outage into silent data loss
discovered only when it is needed.

### Prove it works

```bash
./ops/backup.sh verify
```

Restores the newest snapshot into a scratch database and asserts the ledger
sums to zero — the same invariant the live system checks on every request.
Schedule it nightly:

```bash
sudo cp ops/arcade-verify.{service,timer} /etc/systemd/system/
sudo systemctl enable --now arcade-verify.timer
```

A backup nobody has restored is a rumour.

## Standby

A second machine that continuously restores the newest backup and waits.

```bash
sudo cp ops/arcade-standby.service /etc/systemd/system/
sudo systemctl enable --now arcade-standby
./ops/standby.sh status
```

It restores into a shadow database and swaps names only after verifying the
ledger balances, so the standby is never mid-restore when you need it and never
promotes a corrupt copy.

### Pulling the plug

On the standby:

```bash
./ops/standby.sh promote
```

It fetches the newest backup, verifies the ledger, and starts the arcade. It
does not contact the dead machine, because in the situation this exists for the
dead machine is not answering.

Three things it deliberately does not do, because they are unsafe to automate:

1. **Repoint the endpoint.** DNS or the load balancer's upstream list. Until
   that happens players still reach the dead machine.
2. **Confirm the old machine is down.** Two live instances writing to different
   databases diverge, and the result cannot be merged — both ledgers will be
   internally valid and mutually contradictory.
3. **Let players in before checking `/api/health`.** It must report a zero
   ledger sum.

### What you lose

Up to one backup interval — five minutes of play. Rounds in flight at the
moment of failure are refunded automatically by the reconciler once the
standby is live, because their stakes were debited but never settled.

## Lightning is different

**Do not restore an Alby Hub backup the way you restore the database.**

Lightning channel state is not a snapshot you can roll back. Publishing an old
channel state is interpreted by your counterparty as an attempt to cheat, and
the penalty mechanism can take the entire channel balance. Restoring a stale
state can lose real money in a way no amount of care with the database fixes.

Follow Alby Hub's own backup and recovery procedure. Keep the seed phrase
offline and separate from the machine. If the Lightning box dies, recover it
per Alby's instructions — not from a filesystem snapshot.

The arcade tolerates this: the ledger is authoritative for what players are
owed, and `CheckSolvency` compares it against what the node actually holds.

## Kernel tuning

```bash
./ops/tune-kernel.sh check     # what is below target
sudo ./ops/tune-kernel.sh apply
```

Every value is tied to a measured limit, documented inline. The ones that
matter most:

| Setting | Why |
|---|---|
| `fs.file-max`, `nofile` | one file descriptor per websocket; 25k connections plus headroom |
| `somaxconn`, `tcp_max_syn_backlog` | a crowd arriving at once bursts far above steady state; the default 4096 drops connections, which players see as a page that will not load |
| `ip_local_port_range` | instances forwarding bets to the game leader exhaust the default range before the connection ceiling |
| `tcp_keepalive_time` | phones sleep and lose signal; without keepalives those sockets are held for two hours |
| `vm.swappiness` | swapping a database's working set is worse than reclaiming page cache |

Do not raise the buffer sizes further without a measurement. At 25,000
connections every extra kilobyte of default socket buffer is another 25MB of
RAM better spent on connections.

## Daily checks

```bash
curl -s localhost:8080/api/health | jq          # ledger sums to zero
./ops/standby.sh status                         # standby is current
systemctl status arcade-backup.timer            # backups running
journalctl -u arcade-verify --since yesterday   # last restore test passed
```

The one number that matters is `ledger_sum_msat`. It is zero or the platform
is telling you it is broken.