ca39e8bad9
The costs page states both deductions and, for each game, the maths return alongside what players actually receive. It is rendered from /api/fees, which the server generates from the same schedule it charges, so the published terms cannot drift from the behaviour. Backups every five minutes with a rolling 24 hours. Each dump is checked for size and format before replacing the previous one — a backup script that reports success on a truncated file is worse than none, because it turns a recoverable outage into silent loss found only when needed. A nightly job restores the newest snapshot and asserts the ledger balances. The standby continuously restores into a shadow database and swaps only after verifying the books, so it is never mid-restore when needed and never promotes a corrupt copy. Promotion does not contact the dead machine, and it refuses to start if the ledger does not balance. Kernel tuning is tied to measured limits, not copied defaults. Alby Hub is explicitly excluded from snapshot restore: publishing a stale channel state can lose the channel balance outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
142 lines
5.0 KiB
Markdown
142 lines
5.0 KiB
Markdown
# Operations
|
|
|
|
Persistence, failover, and tuning for a self-hosted arcade.
|
|
|
|
## Machines
|
|
|
|
| Role | Runs | Clone? |
|
|
|---|---|---|
|
|
| **core** | PostgreSQL, Redis, Caddy | no — one only |
|
|
| **app** | `quantum-arcade` | yes, freely |
|
|
| **standby** | PostgreSQL + `standby.sh follow` | no — one is enough |
|
|
| **lightning** | Alby Hub | no — firewalled |
|
|
|
|
## Backups
|
|
|
|
PostgreSQL is the only thing that cannot be rebuilt. The app binary embeds its
|
|
own client, and Redis holds only sessions and leases, which regenerate.
|
|
|
|
```bash
|
|
sudo mkdir -p /var/backups/quantum-arcade
|
|
./ops/backup.sh init # once
|
|
sudo cp ops/arcade-backup.{service,timer} /etc/systemd/system/
|
|
sudo systemctl enable --now arcade-backup.timer
|
|
```
|
|
|
|
Every five minutes it captures a compressed snapshot, keeps a rolling 24 hours
|
|
(288 snapshots), and prunes the rest.
|
|
|
|
**Every dump is checked before it replaces the previous one** — size and format
|
|
header. A backup script that reports success on a truncated file is worse than
|
|
no backup, because it converts a recoverable outage into silent data loss
|
|
discovered only when it is needed.
|
|
|
|
### Prove it works
|
|
|
|
```bash
|
|
./ops/backup.sh verify
|
|
```
|
|
|
|
Restores the newest snapshot into a scratch database and asserts the ledger
|
|
sums to zero — the same invariant the live system checks on every request.
|
|
Schedule it nightly:
|
|
|
|
```bash
|
|
sudo cp ops/arcade-verify.{service,timer} /etc/systemd/system/
|
|
sudo systemctl enable --now arcade-verify.timer
|
|
```
|
|
|
|
A backup nobody has restored is a rumour.
|
|
|
|
## Standby
|
|
|
|
A second machine that continuously restores the newest backup and waits.
|
|
|
|
```bash
|
|
sudo cp ops/arcade-standby.service /etc/systemd/system/
|
|
sudo systemctl enable --now arcade-standby
|
|
./ops/standby.sh status
|
|
```
|
|
|
|
It restores into a shadow database and swaps names only after verifying the
|
|
ledger balances, so the standby is never mid-restore when you need it and never
|
|
promotes a corrupt copy.
|
|
|
|
### Pulling the plug
|
|
|
|
On the standby:
|
|
|
|
```bash
|
|
./ops/standby.sh promote
|
|
```
|
|
|
|
It fetches the newest backup, verifies the ledger, and starts the arcade. It
|
|
does not contact the dead machine, because in the situation this exists for the
|
|
dead machine is not answering.
|
|
|
|
Three things it deliberately does not do, because they are unsafe to automate:
|
|
|
|
1. **Repoint the endpoint.** DNS or the load balancer's upstream list. Until
|
|
that happens players still reach the dead machine.
|
|
2. **Confirm the old machine is down.** Two live instances writing to different
|
|
databases diverge, and the result cannot be merged — both ledgers will be
|
|
internally valid and mutually contradictory.
|
|
3. **Let players in before checking `/api/health`.** It must report a zero
|
|
ledger sum.
|
|
|
|
### What you lose
|
|
|
|
Up to one backup interval — five minutes of play. Rounds in flight at the
|
|
moment of failure are refunded automatically by the reconciler once the
|
|
standby is live, because their stakes were debited but never settled.
|
|
|
|
## Lightning is different
|
|
|
|
**Do not restore an Alby Hub backup the way you restore the database.**
|
|
|
|
Lightning channel state is not a snapshot you can roll back. Publishing an old
|
|
channel state is interpreted by your counterparty as an attempt to cheat, and
|
|
the penalty mechanism can take the entire channel balance. Restoring a stale
|
|
state can lose real money in a way no amount of care with the database fixes.
|
|
|
|
Follow Alby Hub's own backup and recovery procedure. Keep the seed phrase
|
|
offline and separate from the machine. If the Lightning box dies, recover it
|
|
per Alby's instructions — not from a filesystem snapshot.
|
|
|
|
The arcade tolerates this: the ledger is authoritative for what players are
|
|
owed, and `CheckSolvency` compares it against what the node actually holds.
|
|
|
|
## Kernel tuning
|
|
|
|
```bash
|
|
./ops/tune-kernel.sh check # what is below target
|
|
sudo ./ops/tune-kernel.sh apply
|
|
```
|
|
|
|
Every value is tied to a measured limit, documented inline. The ones that
|
|
matter most:
|
|
|
|
| Setting | Why |
|
|
|---|---|
|
|
| `fs.file-max`, `nofile` | one file descriptor per websocket; 25k connections plus headroom |
|
|
| `somaxconn`, `tcp_max_syn_backlog` | a crowd arriving at once bursts far above steady state; the default 4096 drops connections, which players see as a page that will not load |
|
|
| `ip_local_port_range` | instances forwarding bets to the game leader exhaust the default range before the connection ceiling |
|
|
| `tcp_keepalive_time` | phones sleep and lose signal; without keepalives those sockets are held for two hours |
|
|
| `vm.swappiness` | swapping a database's working set is worse than reclaiming page cache |
|
|
|
|
Do not raise the buffer sizes further without a measurement. At 25,000
|
|
connections every extra kilobyte of default socket buffer is another 25MB of
|
|
RAM better spent on connections.
|
|
|
|
## Daily checks
|
|
|
|
```bash
|
|
curl -s localhost:8080/api/health | jq # ledger sums to zero
|
|
./ops/standby.sh status # standby is current
|
|
systemctl status arcade-backup.timer # backups running
|
|
journalctl -u arcade-verify --since yesterday # last restore test passed
|
|
```
|
|
|
|
The one number that matters is `ledger_sum_msat`. It is zero or the platform
|
|
is telling you it is broken.
|