AetherForge/docs/TESTING_GAPS.md

Test coverage gaps

Operator-focused audit of what automation does and does not prove. Counts (2026-06-07): Go server 1007 Test*, agent 680, Vitest 867, Playwright 32 — see tests/README.md. Gates and rungs: VALIDATION_LADDER.md. By-design limits and manual work: PROBLEMS.md.

Headline: CI proves wiring, contracts, and mock-heavy paths. It does not prove live AWS, real spread lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android runtime, or fleet scale in your environment.

Default full gate (scripts/test-suite.ps1 phases 1–8) runs all Go, Vitest, and Playwright. Quick verify (-SkipE2E) drops all 32 Playwright tests — common post-landing smoke, easy to mistake for “release green.” Feature slices (-S3Swarm, -CloudMap, etc.) are focused subsets, not extra coverage beyond go test ./....

---

1. Critical gaps — shipped, little or no real automation

Area	What exists	What's missing
Live AWS	Mock HeadBucket/inject S3 store (s3_swarm_test.go); ZIP exports for SSM, Fargate, policy fan-out; Emberwake Vitest for method catalog	Real PutObject, CloudFront signed magnets, SendCommand, RunTask, operator Lambda/EventBridge (PROBLEMS.md § AWS)
Cloud Map route_via on deploy plans	Agent registry fetch + route_via= in join notes (cloud_map_test.go)	Server AttachCloudMapRouteVia deferred — no wiring tests (PROBLEMS.md)
P2 spread lanes	Template export, lane dispatch, injectable staging (staging_chain_test.go, spread_lanes_test.go)	Live Docker/Podman, WinRM/GPO/systemd/crontab on remote hosts, live BITS/curl (PROBLEMS.md manual only)
Deploy Recon SSRF canary	Go flow with injected dial/fetch hooks + manual CanaryPing (recon_handler_test.go); Playwright mocks POST /recon/scan (deploy-recon.spec.ts)	No automated “paste probe URL into owned target → server fetch → canary confirmed” (PROBLEMS.md)
Subnet recon (live LAN)	-SubnetRecon gate: policy ingest, capped targets, mock port probes (subnet_recon_test.go, subnet_discovery_test.go)	Real ARP/subnet sweep on LAN; service_discover skips live LAN in agent tests (aggressive_commands_test.go)
Live HTTP forge	Builder unit tests	LIVE_FORGE=1 + -tags liveforge only (live_forge_smoke_test.go); e2e-validate.ps1 -ForgeAgent manual
Fleet AI / Court LLM	DecideFunc inject, deterministic contingency persona (persona_mining_test.go, contingency_tree_test.go)	Live Ollama/vLLM; court on every exhaust tick (PROBLEMS.md, tests/README.md Fleet AI gaps)
Android agent	APK builder/config/scout Go tests; android/forge/config_test.go	No emulator/device E2E; no Crucible round-trip for platform=android
Fusion module	Phase 3: go test + go build in fusion/	Only media_crypto_test.go (decrypt round-trip) — no end-to-end fusion bundle + agent launch
Mesh P2P	Stub tests default build (mesh_p2p_stub_test.go)	Real peers need -tags p2p (PROBLEMS.md)
Scale / soak	Some stats_batch coalesce tests	No 500+/1000+ agent soak, SQLITE_BUSY, WS init blob at fleet scale (PROBLEMS.md scale limits)
Activity Feed + ROI	Routes in App.tsx, nav in Layout.tsx	No Vitest, no Playwright — only TSX pages

Stale doc note: PROBLEMS.md still says Path Tracer sessions are RAM-only. Code now has SQLite persist + startup restore (pathtracer_handler.go, pathtracer_persist_test.go). Gap is full process restart E2E and live WireGuard chain, not “no persistence.”

---

2. Partial coverage — mocks/stubs, not real behavior

Spread / mining

• WinRM/GPO/Linux LOTL: script shape and deploy-plan rendering; execution is not run against domain/SSH targets.
• Container mining: SetRuntimeDetector inject; no live docker run / GPU passthrough.
• mining_diagnostics over WS: agent integration uses stubFastMiningDiagnostics (~1s+ probes avoided).
• Discover→spread Playwright: POST discover_and_join + stub WS join_lane: dns_txt — UI wiring only (discover-spread.spec.ts).

Cloud lanes

Each has a -Feature slice in test-suite.ps1; all mock/export in CI:

• S3Swarm: mock shard store, not real bucket policy + signing keys.
• SSM: document + CLI export; no SendCommand.
• Fargate: task-def ZIP + campaign sync; no ECS/Fargate in VPC.
• PolicyFanout: public snapshot URL + fan-out ZIP; relay is operator-deployed.
• CloudVenue: IMDS tag inference via inject; not real g4dn/spot labels on EC2.

Contingency / court

• Agent ContingencyTreeRunner unit-tested (branch order, ghost skip).
• Server pushes contingency_policy when AI control on (contingency_bridge_test.go).
• Court UI: Vitest + Playwright with mocked GET /ai/decisions (lotl-timeline.spec.ts) — not a live stuck-host → scheduler → LLM → L4 dispatch loop.

Path Tracer

• Handler-restart persistence unit test.
• Playwright mocks /pathtrace/start and /status (path-tracer.spec.ts); no wg_setup E2E (agent test skips UPnP: aggressive_commands_test.go).
• Non-Windows agents: pathtracer_stub.go errors on wg_setup.

Deploy Recon

• Port dial and HTTP crawl run on dashboard host (Go server), tested via hooks — not through fleet agents.
• Playwright never hits a real scan; SSRF finding is from fixture JSON.

Subnet recon

• Logic for skip fleet IP, cap hosts, WS ingest — not a lab with multiple live subnets.

Fusion

• Compile + crypto helper only; fusion zip → dropper → agent path is handler unit tests, not a fused-artifact smoke.

---

3. Missing E2E — nav routes vs Playwright (10 specs)

Layout nav (Layout.tsx NAV_BASE + conditional Seer): Command Deck, Crucible, Activity, ROI, Onion, Path Tracer, Forge, Mission Deck, Builds, Emberwake, Deploy Recon, Calibrate; Seer when ai_control_enabled.

Route	Vitest	Playwright
/dashboard	yes	pages.spec.ts, smoke.spec.ts
/crucible	yes	pages.spec.ts, crucible-*.spec.ts, discover-spread.spec.ts, remote-actions.spec.ts
/settings	yes	pages.spec.ts
/forge	yes	pages.spec.ts, smoke.spec.ts
/deploy-recon	yes	pages.spec.ts, deploy-recon.spec.ts (mocked scan)
/emberwake	yes	pages.spec.ts (Cloud ecosystem panel only)
/lotl-timeline	yes	lotl-timeline.spec.ts, crucible-lotl.spec.ts
/pathtracer	yes	path-tracer.spec.ts (mocked API)
/seer, /oath	yes	pages.spec.ts (goto; Oath not in primary nav)
/mission-deck	yes	None
/builds	yes	None
/activity	None	None
/roi	None	None

Emberwake gaps: no Playwright for SSM panel, policy fan-out export, spread template ZIP, launch-template genesis, or connection-test flows beyond “panel expands.”

Contingency: no Playwright at all (only -Contingency Vitest/Go slice).

Court: mocked panel only — not “fleet stuck → live court transcript on WS.”

---

4. Missing integration — multi-hop, cross-service, soak

Gap	Why it matters
Real multi-hop discover→spread	Stub acknowledges 3-hop chain in UI; no agent→agent→target spread
Server process restart + Path Tracer restore	Unit test covers new handler instance; no Playwright “start trace → restart server → resume”
Multi-agent Fleet AI scheduler	One mocked Tick(); no parallel decision races (tests/README.md)
Cross-service cloud deploy	Forge erasure panel → real S3 → agent stage_fetch magnet — never chained in CI
Deploy Recon → Crucible spread	Links exist in UI; no E2E “scan owned host → Probe & Join same target”
Docker mining in default test.bat	Separate GHA ci-docker-mining.yml / scripts/ci-docker-mining.sh — not phase 8
Tier 3 Windows VM playbook	docs/E2E_VALIDATION.md M-01–M-10 checklist — manual
Platform-split agent tests	Linux LOTL/staging tests use //go:build !windows — default Windows go test does not execute them
WS init full-fleet blob	Pagination REST-only; no load test at hundreds of agents
Hashrate retention / purge at scale	Retention jobs exist; no sustained write soak

---

5. Suggested additions (prioritized)

P1 — operator trust / security regressions

1. Deploy Recon SSRF canary live hit — owned httptest target that callbacks /recon/ping/{id}; optional Playwright “copy URL → simulate paste field → assert WS canary_hit.”
2. Activity Feed + ROI — minimal Vitest render + Playwright nav smoke (currently zero coverage).
3. -SkipE2E documentation trap — treat quick verify as “no browser proof”; require phase 8 before release.
4. Cloud Map route_via server wiring — once implemented, deploy-plan test matching agent-side expectation.
5. Path Tracer restart integration — single Go test: persist → NewPathTracerHandler → API status matches (extend beyond handler-only restart).

P2 — shipped features, mock-heavy today

1. Court / contingency live-path smoke — optional OLLAMA_URL job: one stuck snapshot → one court cycle → restart_mining dispatch (skip if no model).
2. Subnet recon LAN fixture — docker-compose mini-LAN or recorded PCAP inject; prove subnet_recon_report → dashboard fleetDiscoveries.
3. Emberwake cloud panels Playwright — SSM export, S3 connection test (mock AWS), policy fan-out download.
4. Mission Deck + Builds — Playwright smoke + one forge→build→download happy path.
5. Linux spread lane CI — run agent deploy/... linux-tagged tests on ubuntu-latest (complement Windows host).
6. Live forge opt-in gate — nightly LIVE_FORGE=1 -tags liveforge job, not PR-blocking.

P3 — scale, polish, deferred architecture

1. Fleet soak harness — N synthetic agents, stats_batch for 30+ min, assert no SQLITE_BUSY / memory growth.
2. Real multi-hop spread lab — second VM in E2E_VALIDATION.md Tier 3 checklist automation.
3. Android emulator smoke — install APK, auth once, scout service_discover.
4. Mesh -tags p2p job — peer count > 0 in isolated network.
5. Path Tracer live WG — lab-only wg_setup with preinstalled WireGuard (skip in CI).
6. Fleet topology 3D — Vitest cap at 200 nodes (PROBLEMS.md); no tests today.

---

Operator takeaway

Passing test.bat means contracts and UI wiring are well guarded. It does not mean AWS spread, lateral lanes, SSRF canary confirmation, LAN subnet discovery, live forge, Android scouts, or fleet scale behave in your environment. Those are explicitly deferred to operator IAM, lab VMs, and E2E_VALIDATION.md Tier 2–3 — which is honest, but easy to forget because test counts look complete.

If you want one command that closest approximates “real” without a Windows VM: ci-docker-mining (Linux C2 + hashrate) plus phase 8 Playwright (stub agents). Everything else in PROBLEMS.md § Manual/live remains on you.

Practical habit after landing:

1. L2 .\scripts\test-suite.ps1 -SkipE2E — fast contract gate.
2. L3 full suite (phase 8) when UI, WS, or routing changed.
3. Read this doc + PROBLEMS.md before claiming production-ready on spread, cloud, or recon.