AetherForge/docs/TESTING_GAPS.md

# Test coverage gaps

Operator-focused audit of what automation **does** and **does not** prove. Counts (2026-06-07): Go server **1007** `Test*`, agent **680**, Vitest **867**, Playwright **32** — see [`tests/README.md`](../tests/README.md). Gates and rungs: [`VALIDATION_LADDER.md`](VALIDATION_LADDER.md). By-design limits and manual work: [`PROBLEMS.md`](../PROBLEMS.md).

**Headline:** CI proves wiring, contracts, and mock-heavy paths. It does **not** prove live AWS, real spread lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android runtime, or fleet scale in your environment.

**Default full gate** (`scripts/test-suite.ps1` phases 1–8) runs all Go, Vitest, and Playwright. **Quick verify** (`-SkipE2E`) drops **all 32 Playwright tests** — common post-landing smoke, easy to mistake for “release green.” Feature slices (`-S3Swarm`, `-CloudMap`, etc.) are focused subsets, not extra coverage beyond `go test ./...`.

---

## 1. Critical gaps — shipped, little or no real automation

| Area | What exists | What's missing |
|------|-------------|----------------|
| **Live AWS** | Mock `HeadBucket`/inject S3 store (`s3_swarm_test.go`); ZIP exports for SSM, Fargate, policy fan-out; Emberwake Vitest for method catalog | Real `PutObject`, CloudFront signed magnets, `SendCommand`, `RunTask`, operator Lambda/EventBridge (`PROBLEMS.md` § AWS) |
| **Cloud Map `route_via` on deploy plans** | Agent registry fetch + `route_via=` in join notes (`cloud_map_test.go`) | Server `AttachCloudMapRouteVia` **deferred** — no wiring tests (`PROBLEMS.md`) |
| **P2 spread lanes** | Template export, lane dispatch, injectable staging (`staging_chain_test.go`, `spread_lanes_test.go`) | Live Docker/Podman, WinRM/GPO/systemd/crontab on remote hosts, live BITS/curl (`PROBLEMS.md` manual only) |
| **Deploy Recon SSRF canary** | Go flow with **injected** dial/fetch hooks + manual `CanaryPing` (`recon_handler_test.go`); Playwright **mocks** `POST /recon/scan` (`deploy-recon.spec.ts`) | No automated “paste probe URL into owned target → server fetch → canary confirmed” (`PROBLEMS.md`) |
| **Subnet recon (live LAN)** | `-SubnetRecon` gate: policy ingest, capped targets, mock port probes (`subnet_recon_test.go`, `subnet_discovery_test.go`) | Real ARP/subnet sweep on LAN; `service_discover` **skips** live LAN in agent tests (`aggressive_commands_test.go`) |
| **Live HTTP forge** | Builder unit tests | `LIVE_FORGE=1` + `-tags liveforge` only (`live_forge_smoke_test.go`); `e2e-validate.ps1 -ForgeAgent` manual |
| **Fleet AI / Court LLM** | `DecideFunc` inject, deterministic contingency persona (`persona_mining_test.go`, `contingency_tree_test.go`) | Live Ollama/vLLM; court on **every** exhaust tick (`PROBLEMS.md`, `tests/README.md` Fleet AI gaps) |
| **Android agent** | APK builder/config/scout Go tests; `android/forge/config_test.go` | No emulator/device E2E; no Crucible round-trip for `platform=android` |
| **Fusion module** | Phase 3: `go test` + `go build` in `fusion/` | Only `media_crypto_test.go` (decrypt round-trip) — no end-to-end fusion bundle + agent launch |
| **Mesh P2P** | Stub tests default build (`mesh_p2p_stub_test.go`) | Real peers need `-tags p2p` (`PROBLEMS.md`) |
| **Scale / soak** | Some `stats_batch` coalesce tests | No 500+/1000+ agent soak, `SQLITE_BUSY`, WS init blob at fleet scale (`PROBLEMS.md` scale limits) |
| **Activity Feed + ROI** | Routes in `App.tsx`, nav in `Layout.tsx` | **No Vitest, no Playwright** — only TSX pages |

**Stale doc note:** `PROBLEMS.md` still says Path Tracer sessions are RAM-only. Code now has SQLite persist + startup restore (`pathtracer_handler.go`, `pathtracer_persist_test.go`). Gap is **full process restart E2E** and **live WireGuard chain**, not “no persistence.”

---

## 2. Partial coverage — mocks/stubs, not real behavior

### Spread / mining

- WinRM/GPO/Linux LOTL: script **shape** and deploy-plan rendering; execution is not run against domain/SSH targets.
- Container mining: `SetRuntimeDetector` inject; no live `docker run` / GPU passthrough.
- `mining_diagnostics` over WS: agent integration uses `stubFastMiningDiagnostics` (~1s+ probes avoided).
- Discover→spread Playwright: POST `discover_and_join` + stub WS `join_lane: dns_txt` — **UI wiring only** (`discover-spread.spec.ts`).

### Cloud lanes

Each has a `-Feature` slice in `test-suite.ps1`; all mock/export in CI:

- **S3Swarm:** mock shard store, not real bucket policy + signing keys.
- **SSM:** document + CLI export; no `SendCommand`.
- **Fargate:** task-def ZIP + campaign sync; no ECS/Fargate in VPC.
- **PolicyFanout:** public snapshot URL + fan-out ZIP; relay is operator-deployed.
- **CloudVenue:** IMDS tag inference via inject; not real `g4dn`/spot labels on EC2.

### Contingency / court

- Agent `ContingencyTreeRunner` unit-tested (branch order, ghost skip).
- Server pushes `contingency_policy` when AI control on (`contingency_bridge_test.go`).
- Court UI: Vitest + Playwright with **mocked** `GET /ai/decisions` (`lotl-timeline.spec.ts`) — not a live stuck-host → scheduler → LLM → L4 dispatch loop.

### Path Tracer

- Handler-restart persistence unit test.
- Playwright **mocks** `/pathtrace/start` and `/status` (`path-tracer.spec.ts`); no `wg_setup` E2E (agent test skips UPnP: `aggressive_commands_test.go`).
- Non-Windows agents: `pathtracer_stub.go` errors on `wg_setup`.

### Deploy Recon

- Port dial and HTTP crawl run on **dashboard host** (Go server), tested via hooks — not through fleet agents.
- Playwright never hits a real scan; SSRF finding is from **fixture JSON**.

### Subnet recon

- Logic for skip fleet IP, cap hosts, WS ingest — not a lab with multiple live subnets.

### Fusion

- Compile + crypto helper only; fusion zip → dropper → agent path is handler unit tests, not a fused-artifact smoke.

---

## 3. Missing E2E — nav routes vs Playwright (10 specs)

**Layout nav** (`Layout.tsx` `NAV_BASE` + conditional Seer): Command Deck, Crucible, Activity, ROI, Onion, Path Tracer, Forge, Mission Deck, Builds, Emberwake, Deploy Recon, Calibrate; Seer when `ai_control_enabled`.

| Route | Vitest | Playwright |
|-------|--------|------------|
| `/dashboard` | yes | `pages.spec.ts`, `smoke.spec.ts` |
| `/crucible` | yes | `pages.spec.ts`, `crucible-*.spec.ts`, `discover-spread.spec.ts`, `remote-actions.spec.ts` |
| `/settings` | yes | `pages.spec.ts` |
| `/forge` | yes | `pages.spec.ts`, `smoke.spec.ts` |
| `/deploy-recon` | yes | `pages.spec.ts`, `deploy-recon.spec.ts` (mocked scan) |
| `/emberwake` | yes | `pages.spec.ts` (Cloud ecosystem panel only) |
| `/lotl-timeline` | yes | `lotl-timeline.spec.ts`, `crucible-lotl.spec.ts` |
| `/pathtracer` | yes | `path-tracer.spec.ts` (mocked API) |
| `/seer`, `/oath` | yes | `pages.spec.ts` (goto; Oath not in primary nav) |
| `/mission-deck` | yes | **None** |
| `/builds` | yes | **None** |
| **`/activity`** | **None** | **None** |
| **`/roi`** | **None** | **None** |

**Emberwake gaps:** no Playwright for SSM panel, policy fan-out export, spread template ZIP, launch-template genesis, or connection-test flows beyond “panel expands.”

**Contingency:** no Playwright at all (only `-Contingency` Vitest/Go slice).

**Court:** mocked panel only — not “fleet stuck → live court transcript on WS.”

---

## 4. Missing integration — multi-hop, cross-service, soak

| Gap | Why it matters |
|-----|----------------|
| **Real multi-hop discover→spread** | Stub acknowledges 3-hop chain in UI; no agent→agent→target spread |
| **Server process restart + Path Tracer restore** | Unit test covers new handler instance; no Playwright “start trace → restart server → resume” |
| **Multi-agent Fleet AI scheduler** | One mocked `Tick()`; no parallel decision races (`tests/README.md`) |
| **Cross-service cloud deploy** | Forge erasure panel → real S3 → agent `stage_fetch` magnet — never chained in CI |
| **Deploy Recon → Crucible spread** | Links exist in UI; no E2E “scan owned host → Probe & Join same target” |
| **Docker mining in default `test.bat`** | Separate GHA `ci-docker-mining.yml` / `scripts/ci-docker-mining.sh` — not phase 8 |
| **Tier 3 Windows VM playbook** | `docs/E2E_VALIDATION.md` M-01–M-10 checklist — manual |
| **Platform-split agent tests** | Linux LOTL/staging tests use `//go:build !windows` — default Windows `go test` does not execute them |
| **WS `init` full-fleet blob** | Pagination REST-only; no load test at hundreds of agents |
| **Hashrate retention / purge at scale** | Retention jobs exist; no sustained write soak |

---

## 5. Suggested additions (prioritized)

### P1 — operator trust / security regressions

1. **Deploy Recon SSRF canary live hit** — owned httptest target that callbacks `/recon/ping/{id}`; optional Playwright “copy URL → simulate paste field → assert WS `canary_hit`.”
2. **Activity Feed + ROI** — minimal Vitest render + Playwright nav smoke (currently zero coverage).
3. **`-SkipE2E` documentation trap** — treat quick verify as “no browser proof”; require phase 8 before release.
4. **Cloud Map `route_via` server wiring** — once implemented, deploy-plan test matching agent-side expectation.
5. **Path Tracer restart integration** — single Go test: persist → `NewPathTracerHandler` → API `status` matches (extend beyond handler-only restart).

### P2 — shipped features, mock-heavy today

1. **Court / contingency live-path smoke** — optional `OLLAMA_URL` job: one stuck snapshot → one court cycle → `restart_mining` dispatch (skip if no model).
2. **Subnet recon LAN fixture** — docker-compose mini-LAN or recorded PCAP inject; prove `subnet_recon_report` → dashboard `fleetDiscoveries`.
3. **Emberwake cloud panels Playwright** — SSM export, S3 connection test (mock AWS), policy fan-out download.
4. **Mission Deck + Builds** — Playwright smoke + one forge→build→download happy path.
5. **Linux spread lane CI** — run agent `deploy/...` linux-tagged tests on `ubuntu-latest` (complement Windows host).
6. **Live forge opt-in gate** — nightly `LIVE_FORGE=1 -tags liveforge` job, not PR-blocking.

### P3 — scale, polish, deferred architecture

1. **Fleet soak harness** — N synthetic agents, `stats_batch` for 30+ min, assert no `SQLITE_BUSY` / memory growth.
2. **Real multi-hop spread lab** — second VM in `E2E_VALIDATION.md` Tier 3 checklist automation.
3. **Android emulator smoke** — install APK, auth once, scout `service_discover`.
4. **Mesh `-tags p2p` job** — peer count > 0 in isolated network.
5. **Path Tracer live WG** — lab-only `wg_setup` with preinstalled WireGuard (skip in CI).
6. **Fleet topology 3D** — Vitest cap at 200 nodes (`PROBLEMS.md`); no tests today.

---

## Operator takeaway

Passing `test.bat` means **contracts and UI wiring are well guarded**. It does **not** mean AWS spread, lateral lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android scouts, or fleet scale behave in your environment. Those are explicitly deferred to **operator IAM, lab VMs, and [`E2E_VALIDATION.md`](E2E_VALIDATION.md) Tier 2–3** — which is honest, but easy to forget because test counts look complete.

If you want one command that closest approximates “real” without a Windows VM: **`ci-docker-mining`** (Linux C2 + hashrate) plus **phase 8 Playwright** (stub agents). Everything else in `PROBLEMS.md` § Manual/live remains on you.

**Practical habit after landing:**

1. **L2** `.\scripts\test-suite.ps1 -SkipE2E` — fast contract gate.
2. **L3** full suite (phase 8) when UI, WS, or routing changed.
3. Read this doc + `PROBLEMS.md` before claiming production-ready on spread, cloud, or recon.