feat: refund abandoned rounds; full-journey and capacity tests

Fixes the money bug flagged earlier. When an instance died mid-round its
players had already been debited, so their stakes sat with the house:
balanced books, quietly robbed players. Every instance now sweeps for
unresolved rounds and refunds them.

Such a round is marked void, not settled. The schema caught this: the
reveal_is_complete constraint requires a settled round to publish its
seed, and an abandoned round has no outcome to reveal. Void is a distinct
state with its own column and a check that the two are exclusive.
Claiming happens before money moves, so concurrent reconcilers on
different instances refund exactly once.

Adds TestFullPlayerJourney: sign-in with no account, fund, scratch, bet
with an auto target, settle, verify the round independently, check the
ledger history is continuous, transfer to a friend, and confirm the books
still sum to zero. It asserts against the ledger rather than the API's
own summary.

Adds cmd/loadtest. One instance on 4 cores held 25,000 concurrent
websocket connections with zero failures at 586MB RSS, about 26KB per
connection, with the load generator competing for the same CPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
drjones
2026-08-05 23:18:29 +00:00
parent c12640cf52
commit 3bdb518f9c
8 changed files with 829 additions and 5 deletions

View File

@@ -78,11 +78,15 @@ t+6s both games taken over, rounds running
Six seconds, unattended. Players attached to the dead instance reconnect
through the load balancer and rejoin whichever instance answers.
The in-flight round on the dead instance is lost — bets already written to the
ledger stand, and the round simply never settles. This is the one rough edge:
stakes are debited at bet time, so a round lost mid-flight leaves those stakes
with the house. A reconciliation job that refunds unsettled rounds is not yet
built.
The in-flight round on the dead instance produces no outcome. Because stakes
are debited when a bet is placed, those players would otherwise be quietly
short — the books stay balanced, but the money sits with the house.
Every instance therefore runs a reconciler every 30 seconds. It finds rounds
left unresolved past a staleness window, marks them **void** (not settled: an
abandoned round has no outcome, so there is no seed to reveal), and refunds
every stake. Claiming the round happens before any money moves, so concurrent
reconcilers on different instances refund exactly once.
## Load balancing
@@ -111,6 +115,31 @@ Returns every registered instance, which one drives each game, and which
instance answered. Useful for confirming a clone joined, and for watching
leadership move during a failover.
## Measured capacity
Run against one instance on a 4-core / 7GB box, with the load generator on the
*same machine* competing for CPU — so these are conservative:
| Connections | Failed | Dial p50 / p99 | Server RSS |
|---|---|---|---|
| 500 | 0 | 1ms / 11ms | — |
| 3,000 | 0 | 1ms / 122ms | — |
| 10,000 | 0 | 1ms / 15ms | 258 MB |
| 25,000 | 0 | 2ms / 1.33s | 586 MB |
About **26KB of server memory per connection**, so 50,000 connections is
roughly 1.2GB — comfortable on any real machine. Connection capacity is not
the constraint people expect it to be.
Reproduce with:
```bash
go run ./cmd/loadtest -conns 10000 -duration 30s -ramp 20s
```
The dial p99 at 25k reflects both processes sharing four cores; on separate
machines it is far lower. Rising dial latency is the signal to add an instance.
## Where this stops scaling
Adding app clones raises the ceiling on connections and fan-out. It does not
@@ -120,6 +149,11 @@ raise these:
cost. Every clone contends for the same database. Getting past this needs
in-memory balance reservation with batched persistence — a change to how
money is held, not a deployment change.
This is the real ceiling, and it is worth being precise about what it means:
50,000 people can *watch* comfortably, and tens of thousands can hold
connections on a single instance. What they cannot all do is place a bet in
the same twenty-second window. A 20s window absorbs roughly 4,600 bets.
- **A single game's round loop** runs on one instance, by design. A game cannot
be split across instances without a distributed clock.