# AetherForge Linux (forge-mesh)

Self-hosted Linux control plane for fleets of enrolled machines. Single Go binary with an embedded React frontend, stratum mining proxy, 14-tier deployment chain, and optional AI-assisted remediation court.

---

## Table of Contents

- [Architecture](#architecture)
- [Components](#components)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Portable USB Deck](#portable-usb-deck)
- [Configuration](#configuration)
- [Building](#building)
- [Installing Agents](#installing-agents)
- [Command Deck (Web UI)](#command-deck-web-ui)
- [API Reference](#api-reference)
- [14-Tier Deployment Chain](#14-tier-deployment-chain)
- [Mining & Stratum Proxy](#mining--stratum-proxy)
- [Singular Machine Court](#singular-machine-court)
- [Alerts (Telegram)](#alerts-telegram)
- [Networking](#networking)
- [systemd Deployment](#systemd-deployment)
- [Testing](#testing)
- [Environment Variables](#environment-variables)
- [Known Limits](#known-limits)
- [Security Notes](#security-notes)
- [Project Layout](#project-layout)

---

## Architecture

```
┌─────────────────────────────────────────────┐
│            forge-mesh-server                │
│  ┌──────────────┐   ┌──────────────────┐    │
│  │  React SPA   │   │   REST / WS API  │    │
│  │  (embedded)  │◄──│   /api/v1/...    │    │
│  └──────────────┘   └────────┬─────────┘    │
│  ┌───────────┐  ┌──────────┐ │ ┌──────────┐ │
│  │  SQLite   │  │  Fleet   │ │ │ Stratum  │ │
│  │   Store   │  │   Hub    │ │ │  Proxy   │ │
│  └───────────┘  └──────────┘ │ └──────────┘ │
└─────────────────────────────┼───────────────┘
                              │ WebSocket / HTTP
           ┌──────────────────┼──────────────┐
           │  forge-mesh-agent (per host)    │
           │  ┌───────────┐  ┌────────────┐  │
           │  │  Mining   │  │  Tier /    │  │
           │  │  Chain    │  │  Beacon    │  │
           │  └───────────┘  └────────────┘  │
           └─────────────────────────────────┘
```

**Server** — serves the embedded SPA, REST+WebSocket API, SQLite state, stratum proxy, and optional AI court.

**Agent** — persistent WebSocket to the deck (falls back to HTTP beacon), tiered mining chain, handles `pause`/`resume`/`reboot`/`screenshot`/`mining_profile` commands, polls for signed self-updates.

---

## Components

| Binary | Purpose |
|--------|---------|
| `forge-mesh-server` | Control plane: API, web UI, SQLite, stratum proxy, court |
| `forge-mesh-agent` | Per-host agent: mining chain, WebSocket heartbeat, command handler |
| `forge-mesh-forge` | CLI: cross-compile and sign agent binaries, record in DB |

---

## Prerequisites

**Required**

| Requirement | Notes |
|-------------|-------|
| Go 1.22+ | `make` targets use `/opt/go/bin/go`; override with `GO=` |
| gcc / build-essential | CGO required for `go-sqlite3` |
| Node.js + npm | Only needed to rebuild the Command Deck frontend |

**Optional** (feature-gated)

| Tool | Feature |
|------|---------|
| `xmrig` | Native CPU miner (simulated fallback if absent) |
| `lolMiner` / `rigel` | GPU mining (KawPoW / RVN) |
| `podman` or `docker` | OCI mining tier (T4, T10) |
| `nix` | Nix flake deploy tier (T11) |
| `wg` / `wireguard-tools` | WireGuard mesh tier (T9) |
| `cloudflared` | Cloudflare tunnel sidecar |
| Ollama / OpenAI-compat API | AI court deliberation |
| `grim`, `scrot`, or `import` | Screenshot command |

---

## Quick Start

```bash
git clone <repo> && cd aetherforge-linux
make all                    # build server + agent + forge CLI
$EDITOR data/config.json    # set credentials and wallet
make run                    # → http://localhost:8989
```

Default login: `admin` / `changeme` (change before exposing to any network).

---

## Portable USB Deck

```bash
# Pack (from repo root after building)
./scripts/pack-usb.sh

# Deploy
tar -xzf dist/forge-mesh-portable-*.tar.gz && cd forge-mesh-portable-*
$EDITOR data/config.json   # change credentials & wallet
./LAUNCH.sh
```

`LAUNCH.sh` checks for a Cloudflare tunnel token, starts `cloudflared` as a sidecar if found, starts `forge-mesh-server`, and opens the deck in your browser.

```bash
AF_NO_BROWSER=1 ./LAUNCH.sh   # headless / SSH
```

---

## Configuration

`data/config.json`:

```json
{
  "listen_addr": ":8989",
  "data_dir": "./data",
  "database_path": "./data/forge-mesh.db",
  "operator_clearance": 4,
  "auth": {
    "basic_username": "admin",
    "basic_password": "changeme",
    "fleet_secret": "change-me-fleet-secret"
  },
  "wallet_policy": {
    "default_wallet": "<XMR wallet address>",
    "currency": "XMR"
  },
  "stratum": {
    "xmr_listen": ":3333",
    "rvn_listen": ":3388",
    "upstream_xmr": "pool.supportxmr.com:3333",
    "upstream_rvn": "stratum-ravencoin.flypool.org:3333"
  },
  "forge": {
    "signing_key_path": "./data/signing.key",
    "artifacts_dir": "./data/artifacts"
  },
  "court": { "ollama_url": "http://127.0.0.1:11434", "enabled": false },
  "telegram": { "enabled": false, "bot_token": "", "chat_id": "" }
}
```

| Key | Description |
|-----|-------------|
| `listen_addr` | HTTP server bind address |
| `operator_clearance` | L0–L4; gates privileged actions |
| `auth.fleet_secret` | Shared secret agents use for Bearer auth |
| `wallet_policy.default_wallet` | Fallback mining wallet address |
| `stratum.*` | Local proxy ports and upstream pool addresses |
| `forge.signing_key_path` | ed25519 key used to sign agent builds |
| `court.ollama_url` | Ollama base URL (requires `court.enabled: true`) |
| `telegram.*` | Bot token + chat ID for fleet event alerts |

---

## Building

```bash
make server       # → bin/forge-mesh-server
make agent        # → bin/forge-mesh-agent
make forge        # → bin/forge-mesh-forge
make all          # build all three
make web-deck     # npm install + build React SPA, sync into webroot
make deck         # web-deck + server
make build        # go build ./... (type-check only)
make tidy         # go mod tidy
```

**Forge CLI** — cross-compile and sign agent artifacts:

```bash
# Build linux/amd64 + linux/arm64, sign ed25519, record in DB
./bin/forge-mesh-forge build --config data/config.json --version 1.2.0

# Print signing public key (pass to agents via -pubkey or FORGE_MESH_PUBKEY)
./bin/forge-mesh-forge pubkey --config data/config.json
```

Signed builds are served at `/api/v1/public/builds` and `/api/v1/public/download/{id}`.

---

## Installing Agents

**One-liner summon:**

```bash
curl -fsSL http://<deck-host>:8989/install.sh | sudo bash
```

The script is rendered from `scripts/install.sh.tpl` with fleet secret, deck URL, and signing public key injected at runtime.

**Manual / systemd:**

```bash
sudo mkdir -p /opt/forge-mesh
sudo install -m 0755 bin/forge-mesh-agent /opt/forge-mesh/agent

sudo tee /etc/forge-mesh/agent.env <<EOF
FORGE_DECK_URL=http://<deck-host>:8989
FORGE_FLEET_SECRET=<your-fleet-secret>
FORGE_PUBKEY=<ed25519-pubkey-hex>
EOF

sudo cp deploy/systemd/forge-mesh-agent.service /etc/systemd/system/
sudo systemctl enable --now forge-mesh-agent
```

**Agent flags:**

| Flag | Env | Description |
|------|-----|-------------|
| `-deck-url` | `FORGE_MESH_DECK_URL` | Deck base URL |
| `-secret` | `FORGE_MESH_FLEET_SECRET` | Fleet bearer secret |
| `-pubkey` | `FORGE_MESH_PUBKEY` | ed25519 public key for signed self-update |
| `-host-id` | — | Stable host ID (persisted to data dir) |
| `-stratum-host` | — | Stratum proxy host (default `127.0.0.1`) |
| `-wallet` | `FORGE_WALLET` | Fallback wallet when no server profile |
| `-data-dir` | — | Agent state dir (default `/opt/forge-mesh` or `~/.forge-mesh`) |

---

## Command Deck (Web UI)

Develop locally:

```bash
cd web && npm install && npm run dev   # proxies /api → http://localhost:8989
```

| Route | Description |
|-------|-------------|
| `/login` | HTTP Basic login (credentials in `sessionStorage`) |
| `/dashboard` | Fleet host cards, real-time hashrate, tier badges, WS health |
| `/forge` | Trigger cross-compiled agent builds |
| `/calibrate` | Mining profile policy editor — push profiles to fleet |
| `/crucible` | Batch terminal — dispatch commands to enrolled hosts |
| `/seer` | SSE event stream — live audit log of fleet and court events |

---

## API Reference

Protected endpoints require HTTP Basic auth. Agent endpoints use `Authorization: Bearer <fleet-secret>`.

### Public

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/health` | Version + health check |
| `GET` | `/install.sh` | Rendered agent install script |
| `GET` | `/api/v1/public/builds` | List public builds |
| `GET` | `/api/v1/public/builds/latest` | Latest public build |
| `GET` | `/api/v1/public/download/{id}` | Download a signed artifact |

### Agent (fleet secret)

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/v1/fleet/register` | Register host, returns `host_id` |
| `POST` | `/api/v1/fleet/beacon` | HTTP fallback heartbeat |
| `GET` | `/api/v1/ws/fleet` | WebSocket: heartbeat + command delivery |

### Protected (operator)

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/fleet` | List all enrolled hosts |
| `POST` | `/api/v1/fleet/{id}/command` | Dispatch a command |
| `POST` | `/api/v1/fleet/{id}/mining-profile` | Push a mining profile |
| `GET` | `/api/v1/fleet/{id}/lotl/timeline` | LOTL tier attempt history |
| `POST` | `/api/v1/fleet/{id}/lotl/run` | Trigger adaptive tier run |
| `GET` | `/api/v1/fleet/{id}/spread-gate` | Earn-before-spread gate status |
| `GET/PUT` | `/api/v1/policy/wallet` | Get / update wallet policy |
| `GET/PUT` | `/api/v1/policy/mining-profile` | Get / set global mining profile |
| `POST` | `/api/v1/policy/snapshot` | Create shareable policy snapshot token |
| `GET` | `/api/v1/forge/builds` | List all recorded builds |
| `POST` | `/api/v1/forge/builds/trigger` | Trigger build pipeline |
| `POST` | `/api/v1/crucible/dispatch` | Dispatch batch terminal command |
| `GET` | `/api/v1/crucible/history` | Crucible command history |
| `GET` | `/api/v1/seer` | SSE stream of seer events |
| `POST` | `/api/v1/court/sessions` | Open a court session |
| `POST` | `/api/v1/court/sessions/{id}/deliberate` | Run prosecutor/defender/judge |
| `POST` | `/api/v1/court/sessions/{id}/verdict` | Dispatch final verdict |
| `GET/POST` | `/api/v1/wireguard/peers` | List or add WireGuard peers |
| `GET` | `/api/v1/wireguard/config` | Render WireGuard config |

---

## 14-Tier Deployment Chain

Sequential fallback chain. Each tier is gated by environment recon and a patch-first check. The Atlas skip system learns which tiers fail per host phenotype and skips them on retry.

| # | Type | Description |
|---|------|-------------|
| T1 | `ssh_key` | SSH key-based push |
| T2 | `curl_bash` | `curl \| bash` summon |
| T3 | `ansible_pull` | Ansible pull from control repo |
| T4 | `podman_rootless` | Rootless Podman container |
| T5 | `systemd_transient` | `systemd-run` transient unit |
| T6 | `snap_flatpak` | Snap or Flatpak package |
| T7 | `lan_cache_peer` | LAN peer cache distribution |
| T8 | `dns_txt` | DNS TXT record bootstrap |
| T9 | `mtls_wireguard` | mTLS mesh join over WireGuard |
| T10 | `immutable_oci` | Immutable OCI image (Docker/Podman) |
| T11 | `nix_flake` | Nix flake derivation |
| T12 | `erasure_reassembly` | Reed-Solomon erasure-coded shard reassembly |
| T13 | `fleet_torrent` | Fleet-seeded torrent distribution |
| T14 | `offline_contingency` | Offline/USB bundle fallback |

The adaptive engine reorders tiers by historical success rate per phenotype on subsequent runs.

---

## Mining & Stratum Proxy

Built-in stratum TCP proxy relays agent miner connections to upstream pools. Miners are tried in priority order; first success becomes active.

| Miner | Algo | Notes |
|-------|------|-------|
| `xmrig` | RandomX (XMR) | Native binary; simulated fallback if absent |
| GPU (lolMiner/Rigel) | KawPoW (RVN) | Requires `nvidia-smi` or `rocm-smi` |
| OCI (podman/docker) | RandomX | Pulls `xmrig/xmrig` via Podman |
| Stratum (raw TCP) | XMR / KawPoW | Validates deck proxy path |

If no miner binary is found, a mock miner keeps the chain alive with simulated hashrate. GPU tiers require proprietary drivers installed separately.

---

## Singular Machine Court

Optional AI-assisted triage for stuck or failing hosts. Requires `court.enabled: true` and a running [Ollama](https://ollama.ai) instance (or any OpenAI-compatible API).

1. **Open** — `POST /api/v1/court/sessions` starts a case for a host.
2. **Deliberate** — gathers LOTL tier evidence and runs three `llama3.2` prompts:
   - **Prosecutor**: argues for aggressive remediation.
   - **Defender**: argues for conservative retry.
   - **Judge**: synthesises a concise operational verdict.
3. **Verdict** — L4 clearance operator approves; supported actions:
   - `retry_adaptive_tiers` — re-runs the deploy chain with adaptive ordering.
   - `pause_host` — marks the host paused.

All transcripts and verdicts persist in SQLite and emit as Seer events. Without Ollama, court runs in stub mode with placeholder responses.

---

## Alerts (Telegram)

```json
"telegram": { "enabled": true, "bot_token": "TOKEN", "chat_id": "CHAT_ID" }
```

Fires on fleet commands dispatched, court verdicts, and significant fleet events.

---

## Networking

### Cloudflare Tunnel

`cloudflared` runs as an external sidecar (not in-process). Provide a token and `LAUNCH.sh` handles the rest:

```bash
echo 'YOUR_CF_TUNNEL_TOKEN' > data/cloudflared-token.txt
./LAUNCH.sh
```

Or set `AF_TUNNEL_TOKEN` in the environment. `LAUNCH.sh` looks for `cloudflared` on `PATH` or in `bin/cloudflared`.

### WireGuard Mesh (operator-managed)

AetherForge does **not** auto-provision WireGuard. Configure it yourself:

```bash
apt install wireguard wireguard-tools
wg genkey | tee data/wg-private.key | wg pubkey > data/wg-public.key
```

`/etc/wireguard/forge-mesh.conf`:

```ini
[Interface]
PrivateKey = <deck-private-key>
Address = 10.66.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <agent-public-key>
AllowedIPs = 10.66.0.2/32
```

```bash
sudo systemctl enable --now wg-quick@forge-mesh
```

Point agents at the deck WireGuard IP: `FORGE_MESH_DECK_URL=http://10.66.0.1:8989`. Peer records are managed via `/api/v1/wireguard/peers`.

---

## systemd Deployment

```bash
# Control plane
sudo cp deploy/systemd/forge-mesh-server.service /etc/systemd/system/
sudo mkdir -p /opt/forge-mesh && sudo cp -r bin/ data/ /opt/forge-mesh/
sudo systemctl enable --now forge-mesh-server

# Agent (on enrolled hosts)
sudo cp deploy/systemd/forge-mesh-agent.service /etc/systemd/system/
sudo tee /etc/forge-mesh/agent.env <<EOF
FORGE_DECK_URL=http://<deck-ip>:8989
FORGE_FLEET_SECRET=<fleet-secret>
FORGE_PUBKEY=<pubkey-hex>
EOF
sudo systemctl enable --now forge-mesh-agent
```

Server working directory: `/opt/forge-mesh`. Config: `/opt/forge-mesh/data/config.json`.

---

## Testing

```bash
# Backend
make test                    # or: go test ./...

# Frontend
cd web && npm install
npm run test                 # run once
npm run test:watch           # watch mode
npm run test:coverage        # V8 coverage

# From repo root
make test-frontend

# End-to-end
make e2e                     # or: ./scripts/e2e-test.sh
```

See [`docs/TESTING.md`](docs/TESTING.md) for MSW handlers, mock WebSocket/EventSource, and how to write new tests.

---

## Environment Variables

| Variable | Scope | Description |
|----------|-------|-------------|
| `FORGE_MESH_DECK_URL` | Agent | Deck base URL |
| `FORGE_MESH_FLEET_SECRET` | Agent | Fleet bearer secret |
| `FORGE_MESH_PUBKEY` | Agent | ed25519 public key hex for signed self-update |
| `FORGE_WALLET` | Agent | Fallback mining wallet |
| `AF_TUNNEL_TOKEN` | Launcher | Cloudflare tunnel token (overrides file) |
| `AF_TUNNEL_EXTERNAL` | Launcher | Set to `1` automatically when sidecar starts |
| `AF_NO_BROWSER` | Launcher | Skip opening a browser |
| `AF_CONFIG` | Launcher | Override config path |
| `AF_DATA_DIR` | Launcher | Data directory path |
| `AF_SERVER_BIN` | Launcher | Path to `forge-mesh-server` binary |
| `AF_DECK_URL` | Launcher | Browser URL (default `http://localhost:8989`) |

---

## Known Limits

See [`docs/PROBLEMS.md`](docs/PROBLEMS.md) for the full list.

- **No worm-style LAN spread.** Enrolled-only; gated by earn-before-burn phenotype check.
- **WireGuard is operator-managed.** No auto-provisioning of keys, configs, or NAT traversal.
- **GPU tiers need drivers pre-installed.** The summon installer does not install GPU stacks.
- **Court requires an LLM you operate.** No bundled model.
- **SQLite is single-node.** Multi-deck HA and Postgres are out of scope.
- **Signature verify is dev-grade.** Dev builds may skip verification with a warning.
- **USB copies are not hardened live media.** Rotate all secrets before production use.

---

## Security Notes

- Change `auth.basic_password` and `auth.fleet_secret` before exposing the deck.
- Do not commit `data/signing.key`, `data/cloudflared-token.txt`, or `data/forge-mesh.db`.
- L0–L4 clearance gates are policy-enforced, not cryptographic. Compromised deck credentials bypass all gates.
- USB copies carry your fleet secret — treat the device like a key.
- Subnet sweeps (`/api/v1/fleet/subnets/sweep`) only probe operator-declared CIDRs.

---

## Project Layout

```
aetherforge-linux/
├── cmd/
│   ├── agent/          # forge-mesh-agent binary
│   ├── forge/          # forge CLI (build/sign artifacts)
│   └── server/         # forge-mesh-server + embedded webroot
├── internal/
│   ├── alerts/         # Telegram notifier
│   ├── api/            # HTTP server, router, handlers, WS hub
│   ├── auth/           # Basic auth, fleet secret middleware, ticket store
│   ├── config/         # Config loader
│   ├── court/          # Singular Machine Court + Seer event hub
│   ├── db/             # SQLite open/migrate
│   ├── erasure/        # Reed-Solomon erasure coding (T12)
│   ├── fleet/          # Host store, fleet hub, tier chain, recon, atlas
│   ├── forge/          # Build pipeline, key pair, artifact signing
│   ├── mining/         # Mining chain, tier implementations, mock miner
│   ├── policy/         # Mining profile policy
│   ├── stratum/        # TCP stratum proxy (XMR + RVN)
│   └── testutil/       # Shared test helpers
├── web/                # React + Vite + Tailwind Command Deck
├── deploy/systemd/     # systemd unit files
├── scripts/            # LAUNCH.sh, pack-usb.sh, test runners, install template
├── data/               # Runtime data (config, SQLite, signing key, artifacts)
├── docs/               # TESTING.md, PROBLEMS.md
├── dist/               # Packed portable tarballs
└── Makefile
```