feat: refund abandoned rounds; full-journey and capacity tests
Fixes the money bug flagged earlier. When an instance died mid-round its players had already been debited, so their stakes sat with the house: balanced books, quietly robbed players. Every instance now sweeps for unresolved rounds and refunds them. Such a round is marked void, not settled. The schema caught this: the reveal_is_complete constraint requires a settled round to publish its seed, and an abandoned round has no outcome to reveal. Void is a distinct state with its own column and a check that the two are exclusive. Claiming happens before money moves, so concurrent reconcilers on different instances refund exactly once. Adds TestFullPlayerJourney: sign-in with no account, fund, scratch, bet with an auto target, settle, verify the round independently, check the ledger history is continuous, transfer to a friend, and confirm the books still sum to zero. It asserts against the ledger rather than the API's own summary. Adds cmd/loadtest. One instance on 4 cores held 25,000 concurrent websocket connections with zero failures at 586MB RSS, about 26KB per connection, with the load generator competing for the same CPU. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -78,11 +78,15 @@ t+6s both games taken over, rounds running
|
||||
Six seconds, unattended. Players attached to the dead instance reconnect
|
||||
through the load balancer and rejoin whichever instance answers.
|
||||
|
||||
The in-flight round on the dead instance is lost — bets already written to the
|
||||
ledger stand, and the round simply never settles. This is the one rough edge:
|
||||
stakes are debited at bet time, so a round lost mid-flight leaves those stakes
|
||||
with the house. A reconciliation job that refunds unsettled rounds is not yet
|
||||
built.
|
||||
The in-flight round on the dead instance produces no outcome. Because stakes
|
||||
are debited when a bet is placed, those players would otherwise be quietly
|
||||
short — the books stay balanced, but the money sits with the house.
|
||||
|
||||
Every instance therefore runs a reconciler every 30 seconds. It finds rounds
|
||||
left unresolved past a staleness window, marks them **void** (not settled: an
|
||||
abandoned round has no outcome, so there is no seed to reveal), and refunds
|
||||
every stake. Claiming the round happens before any money moves, so concurrent
|
||||
reconcilers on different instances refund exactly once.
|
||||
|
||||
## Load balancing
|
||||
|
||||
@@ -111,6 +115,31 @@ Returns every registered instance, which one drives each game, and which
|
||||
instance answered. Useful for confirming a clone joined, and for watching
|
||||
leadership move during a failover.
|
||||
|
||||
## Measured capacity
|
||||
|
||||
Run against one instance on a 4-core / 7GB box, with the load generator on the
|
||||
*same machine* competing for CPU — so these are conservative:
|
||||
|
||||
| Connections | Failed | Dial p50 / p99 | Server RSS |
|
||||
|---|---|---|---|
|
||||
| 500 | 0 | 1ms / 11ms | — |
|
||||
| 3,000 | 0 | 1ms / 122ms | — |
|
||||
| 10,000 | 0 | 1ms / 15ms | 258 MB |
|
||||
| 25,000 | 0 | 2ms / 1.33s | 586 MB |
|
||||
|
||||
About **26KB of server memory per connection**, so 50,000 connections is
|
||||
roughly 1.2GB — comfortable on any real machine. Connection capacity is not
|
||||
the constraint people expect it to be.
|
||||
|
||||
Reproduce with:
|
||||
|
||||
```bash
|
||||
go run ./cmd/loadtest -conns 10000 -duration 30s -ramp 20s
|
||||
```
|
||||
|
||||
The dial p99 at 25k reflects both processes sharing four cores; on separate
|
||||
machines it is far lower. Rising dial latency is the signal to add an instance.
|
||||
|
||||
## Where this stops scaling
|
||||
|
||||
Adding app clones raises the ceiling on connections and fan-out. It does not
|
||||
@@ -120,6 +149,11 @@ raise these:
|
||||
cost. Every clone contends for the same database. Getting past this needs
|
||||
in-memory balance reservation with batched persistence — a change to how
|
||||
money is held, not a deployment change.
|
||||
|
||||
This is the real ceiling, and it is worth being precise about what it means:
|
||||
50,000 people can *watch* comfortably, and tens of thousands can hold
|
||||
connections on a single instance. What they cannot all do is place a bet in
|
||||
the same twenty-second window. A 20s window absorbs roughly 4,600 bets.
|
||||
- **A single game's round loop** runs on one instance, by design. A game cannot
|
||||
be split across instances without a distributed clock.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user