Files
LINUX-AETHERFORGE/README.md
drjones 3678b199d0
Some checks failed
Test / test (push) Has been cancelled
Initial commit: AetherForge Linux (forge-mesh) v0.1.0-dev
2026-07-04 09:31:23 +00:00

539 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AetherForge Linux (forge-mesh)
Self-hosted Linux control plane for fleets of enrolled machines. Single Go binary with an embedded React frontend, stratum mining proxy, 14-tier deployment chain, and optional AI-assisted remediation court.
---
## Table of Contents
- [Architecture](#architecture)
- [Components](#components)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Portable USB Deck](#portable-usb-deck)
- [Configuration](#configuration)
- [Building](#building)
- [Installing Agents](#installing-agents)
- [Command Deck (Web UI)](#command-deck-web-ui)
- [API Reference](#api-reference)
- [14-Tier Deployment Chain](#14-tier-deployment-chain)
- [Mining & Stratum Proxy](#mining--stratum-proxy)
- [Singular Machine Court](#singular-machine-court)
- [Alerts (Telegram)](#alerts-telegram)
- [Networking](#networking)
- [systemd Deployment](#systemd-deployment)
- [Testing](#testing)
- [Environment Variables](#environment-variables)
- [Known Limits](#known-limits)
- [Security Notes](#security-notes)
- [Project Layout](#project-layout)
---
## Architecture
```
┌─────────────────────────────────────────────┐
│ forge-mesh-server │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ React SPA │ │ REST / WS API │ │
│ │ (embedded) │◄──│ /api/v1/... │ │
│ └──────────────┘ └────────┬─────────┘ │
│ ┌───────────┐ ┌──────────┐ │ ┌──────────┐ │
│ │ SQLite │ │ Fleet │ │ │ Stratum │ │
│ │ Store │ │ Hub │ │ │ Proxy │ │
│ └───────────┘ └──────────┘ │ └──────────┘ │
└─────────────────────────────┼───────────────┘
│ WebSocket / HTTP
┌──────────────────┼──────────────┐
│ forge-mesh-agent (per host) │
│ ┌───────────┐ ┌────────────┐ │
│ │ Mining │ │ Tier / │ │
│ │ Chain │ │ Beacon │ │
│ └───────────┘ └────────────┘ │
└─────────────────────────────────┘
```
**Server** — serves the embedded SPA, REST+WebSocket API, SQLite state, stratum proxy, and optional AI court.
**Agent** — persistent WebSocket to the deck (falls back to HTTP beacon), tiered mining chain, handles `pause`/`resume`/`reboot`/`screenshot`/`mining_profile` commands, polls for signed self-updates.
---
## Components
| Binary | Purpose |
|--------|---------|
| `forge-mesh-server` | Control plane: API, web UI, SQLite, stratum proxy, court |
| `forge-mesh-agent` | Per-host agent: mining chain, WebSocket heartbeat, command handler |
| `forge-mesh-forge` | CLI: cross-compile and sign agent binaries, record in DB |
---
## Prerequisites
**Required**
| Requirement | Notes |
|-------------|-------|
| Go 1.22+ | `make` targets use `/opt/go/bin/go`; override with `GO=` |
| gcc / build-essential | CGO required for `go-sqlite3` |
| Node.js + npm | Only needed to rebuild the Command Deck frontend |
**Optional** (feature-gated)
| Tool | Feature |
|------|---------|
| `xmrig` | Native CPU miner (simulated fallback if absent) |
| `lolMiner` / `rigel` | GPU mining (KawPoW / RVN) |
| `podman` or `docker` | OCI mining tier (T4, T10) |
| `nix` | Nix flake deploy tier (T11) |
| `wg` / `wireguard-tools` | WireGuard mesh tier (T9) |
| `cloudflared` | Cloudflare tunnel sidecar |
| Ollama / OpenAI-compat API | AI court deliberation |
| `grim`, `scrot`, or `import` | Screenshot command |
---
## Quick Start
```bash
git clone <repo> && cd aetherforge-linux
make all # build server + agent + forge CLI
$EDITOR data/config.json # set credentials and wallet
make run # → http://localhost:8989
```
Default login: `admin` / `changeme` (change before exposing to any network).
---
## Portable USB Deck
```bash
# Pack (from repo root after building)
./scripts/pack-usb.sh
# Deploy
tar -xzf dist/forge-mesh-portable-*.tar.gz && cd forge-mesh-portable-*
$EDITOR data/config.json # change credentials & wallet
./LAUNCH.sh
```
`LAUNCH.sh` checks for a Cloudflare tunnel token, starts `cloudflared` as a sidecar if found, starts `forge-mesh-server`, and opens the deck in your browser.
```bash
AF_NO_BROWSER=1 ./LAUNCH.sh # headless / SSH
```
---
## Configuration
`data/config.json`:
```json
{
"listen_addr": ":8989",
"data_dir": "./data",
"database_path": "./data/forge-mesh.db",
"operator_clearance": 4,
"auth": {
"basic_username": "admin",
"basic_password": "changeme",
"fleet_secret": "change-me-fleet-secret"
},
"wallet_policy": {
"default_wallet": "<XMR wallet address>",
"currency": "XMR"
},
"stratum": {
"xmr_listen": ":3333",
"rvn_listen": ":3388",
"upstream_xmr": "pool.supportxmr.com:3333",
"upstream_rvn": "stratum-ravencoin.flypool.org:3333"
},
"forge": {
"signing_key_path": "./data/signing.key",
"artifacts_dir": "./data/artifacts"
},
"court": { "ollama_url": "http://127.0.0.1:11434", "enabled": false },
"telegram": { "enabled": false, "bot_token": "", "chat_id": "" }
}
```
| Key | Description |
|-----|-------------|
| `listen_addr` | HTTP server bind address |
| `operator_clearance` | L0–L4; gates privileged actions |
| `auth.fleet_secret` | Shared secret agents use for Bearer auth |
| `wallet_policy.default_wallet` | Fallback mining wallet address |
| `stratum.*` | Local proxy ports and upstream pool addresses |
| `forge.signing_key_path` | ed25519 key used to sign agent builds |
| `court.ollama_url` | Ollama base URL (requires `court.enabled: true`) |
| `telegram.*` | Bot token + chat ID for fleet event alerts |
---
## Building
```bash
make server # → bin/forge-mesh-server
make agent # → bin/forge-mesh-agent
make forge # → bin/forge-mesh-forge
make all # build all three
make web-deck # npm install + build React SPA, sync into webroot
make deck # web-deck + server
make build # go build ./... (type-check only)
make tidy # go mod tidy
```
**Forge CLI** — cross-compile and sign agent artifacts:
```bash
# Build linux/amd64 + linux/arm64, sign ed25519, record in DB
./bin/forge-mesh-forge build --config data/config.json --version 1.2.0
# Print signing public key (pass to agents via -pubkey or FORGE_MESH_PUBKEY)
./bin/forge-mesh-forge pubkey --config data/config.json
```
Signed builds are served at `/api/v1/public/builds` and `/api/v1/public/download/{id}`.
---
## Installing Agents
**One-liner summon:**
```bash
curl -fsSL http://<deck-host>:8989/install.sh | sudo bash
```
The script is rendered from `scripts/install.sh.tpl` with fleet secret, deck URL, and signing public key injected at runtime.
**Manual / systemd:**
```bash
sudo mkdir -p /opt/forge-mesh
sudo install -m 0755 bin/forge-mesh-agent /opt/forge-mesh/agent
sudo tee /etc/forge-mesh/agent.env <<EOF
FORGE_DECK_URL=http://<deck-host>:8989
FORGE_FLEET_SECRET=<your-fleet-secret>
FORGE_PUBKEY=<ed25519-pubkey-hex>
EOF
sudo cp deploy/systemd/forge-mesh-agent.service /etc/systemd/system/
sudo systemctl enable --now forge-mesh-agent
```
**Agent flags:**
| Flag | Env | Description |
|------|-----|-------------|
| `-deck-url` | `FORGE_MESH_DECK_URL` | Deck base URL |
| `-secret` | `FORGE_MESH_FLEET_SECRET` | Fleet bearer secret |
| `-pubkey` | `FORGE_MESH_PUBKEY` | ed25519 public key for signed self-update |
| `-host-id` | — | Stable host ID (persisted to data dir) |
| `-stratum-host` | — | Stratum proxy host (default `127.0.0.1`) |
| `-wallet` | `FORGE_WALLET` | Fallback wallet when no server profile |
| `-data-dir` | — | Agent state dir (default `/opt/forge-mesh` or `~/.forge-mesh`) |
---
## Command Deck (Web UI)
Develop locally:
```bash
cd web && npm install && npm run dev # proxies /api → http://localhost:8989
```
| Route | Description |
|-------|-------------|
| `/login` | HTTP Basic login (credentials in `sessionStorage`) |
| `/dashboard` | Fleet host cards, real-time hashrate, tier badges, WS health |
| `/forge` | Trigger cross-compiled agent builds |
| `/calibrate` | Mining profile policy editor — push profiles to fleet |
| `/crucible` | Batch terminal — dispatch commands to enrolled hosts |
| `/seer` | SSE event stream — live audit log of fleet and court events |
---
## API Reference
Protected endpoints require HTTP Basic auth. Agent endpoints use `Authorization: Bearer <fleet-secret>`.
### Public
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/health` | Version + health check |
| `GET` | `/install.sh` | Rendered agent install script |
| `GET` | `/api/v1/public/builds` | List public builds |
| `GET` | `/api/v1/public/builds/latest` | Latest public build |
| `GET` | `/api/v1/public/download/{id}` | Download a signed artifact |
### Agent (fleet secret)
| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/v1/fleet/register` | Register host, returns `host_id` |
| `POST` | `/api/v1/fleet/beacon` | HTTP fallback heartbeat |
| `GET` | `/api/v1/ws/fleet` | WebSocket: heartbeat + command delivery |
### Protected (operator)
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/fleet` | List all enrolled hosts |
| `POST` | `/api/v1/fleet/{id}/command` | Dispatch a command |
| `POST` | `/api/v1/fleet/{id}/mining-profile` | Push a mining profile |
| `GET` | `/api/v1/fleet/{id}/lotl/timeline` | LOTL tier attempt history |
| `POST` | `/api/v1/fleet/{id}/lotl/run` | Trigger adaptive tier run |
| `GET` | `/api/v1/fleet/{id}/spread-gate` | Earn-before-spread gate status |
| `GET/PUT` | `/api/v1/policy/wallet` | Get / update wallet policy |
| `GET/PUT` | `/api/v1/policy/mining-profile` | Get / set global mining profile |
| `POST` | `/api/v1/policy/snapshot` | Create shareable policy snapshot token |
| `GET` | `/api/v1/forge/builds` | List all recorded builds |
| `POST` | `/api/v1/forge/builds/trigger` | Trigger build pipeline |
| `POST` | `/api/v1/crucible/dispatch` | Dispatch batch terminal command |
| `GET` | `/api/v1/crucible/history` | Crucible command history |
| `GET` | `/api/v1/seer` | SSE stream of seer events |
| `POST` | `/api/v1/court/sessions` | Open a court session |
| `POST` | `/api/v1/court/sessions/{id}/deliberate` | Run prosecutor/defender/judge |
| `POST` | `/api/v1/court/sessions/{id}/verdict` | Dispatch final verdict |
| `GET/POST` | `/api/v1/wireguard/peers` | List or add WireGuard peers |
| `GET` | `/api/v1/wireguard/config` | Render WireGuard config |
---
## 14-Tier Deployment Chain
Sequential fallback chain. Each tier is gated by environment recon and a patch-first check. The Atlas skip system learns which tiers fail per host phenotype and skips them on retry.
| # | Type | Description |
|---|------|-------------|
| T1 | `ssh_key` | SSH key-based push |
| T2 | `curl_bash` | `curl \| bash` summon |
| T3 | `ansible_pull` | Ansible pull from control repo |
| T4 | `podman_rootless` | Rootless Podman container |
| T5 | `systemd_transient` | `systemd-run` transient unit |
| T6 | `snap_flatpak` | Snap or Flatpak package |
| T7 | `lan_cache_peer` | LAN peer cache distribution |
| T8 | `dns_txt` | DNS TXT record bootstrap |
| T9 | `mtls_wireguard` | mTLS mesh join over WireGuard |
| T10 | `immutable_oci` | Immutable OCI image (Docker/Podman) |
| T11 | `nix_flake` | Nix flake derivation |
| T12 | `erasure_reassembly` | Reed-Solomon erasure-coded shard reassembly |
| T13 | `fleet_torrent` | Fleet-seeded torrent distribution |
| T14 | `offline_contingency` | Offline/USB bundle fallback |
The adaptive engine reorders tiers by historical success rate per phenotype on subsequent runs.
---
## Mining & Stratum Proxy
Built-in stratum TCP proxy relays agent miner connections to upstream pools. Miners are tried in priority order; first success becomes active.
| Miner | Algo | Notes |
|-------|------|-------|
| `xmrig` | RandomX (XMR) | Native binary; simulated fallback if absent |
| GPU (lolMiner/Rigel) | KawPoW (RVN) | Requires `nvidia-smi` or `rocm-smi` |
| OCI (podman/docker) | RandomX | Pulls `xmrig/xmrig` via Podman |
| Stratum (raw TCP) | XMR / KawPoW | Validates deck proxy path |
If no miner binary is found, a mock miner keeps the chain alive with simulated hashrate. GPU tiers require proprietary drivers installed separately.
---
## Singular Machine Court
Optional AI-assisted triage for stuck or failing hosts. Requires `court.enabled: true` and a running [Ollama](https://ollama.ai) instance (or any OpenAI-compatible API).
1. **Open** — `POST /api/v1/court/sessions` starts a case for a host.
2. **Deliberate** — gathers LOTL tier evidence and runs three `llama3.2` prompts:
- **Prosecutor**: argues for aggressive remediation.
- **Defender**: argues for conservative retry.
- **Judge**: synthesises a concise operational verdict.
3. **Verdict** — L4 clearance operator approves; supported actions:
- `retry_adaptive_tiers` — re-runs the deploy chain with adaptive ordering.
- `pause_host` — marks the host paused.
All transcripts and verdicts persist in SQLite and emit as Seer events. Without Ollama, court runs in stub mode with placeholder responses.
---
## Alerts (Telegram)
```json
"telegram": { "enabled": true, "bot_token": "TOKEN", "chat_id": "CHAT_ID" }
```
Fires on fleet commands dispatched, court verdicts, and significant fleet events.
---
## Networking
### Cloudflare Tunnel
`cloudflared` runs as an external sidecar (not in-process). Provide a token and `LAUNCH.sh` handles the rest:
```bash
echo 'YOUR_CF_TUNNEL_TOKEN' > data/cloudflared-token.txt
./LAUNCH.sh
```
Or set `AF_TUNNEL_TOKEN` in the environment. `LAUNCH.sh` looks for `cloudflared` on `PATH` or in `bin/cloudflared`.
### WireGuard Mesh (operator-managed)
AetherForge does **not** auto-provision WireGuard. Configure it yourself:
```bash
apt install wireguard wireguard-tools
wg genkey | tee data/wg-private.key | wg pubkey > data/wg-public.key
```
`/etc/wireguard/forge-mesh.conf`:
```ini
[Interface]
PrivateKey = <deck-private-key>
Address = 10.66.0.1/24
ListenPort = 51820
[Peer]
PublicKey = <agent-public-key>
AllowedIPs = 10.66.0.2/32
```
```bash
sudo systemctl enable --now wg-quick@forge-mesh
```
Point agents at the deck WireGuard IP: `FORGE_MESH_DECK_URL=http://10.66.0.1:8989`. Peer records are managed via `/api/v1/wireguard/peers`.
---
## systemd Deployment
```bash
# Control plane
sudo cp deploy/systemd/forge-mesh-server.service /etc/systemd/system/
sudo mkdir -p /opt/forge-mesh && sudo cp -r bin/ data/ /opt/forge-mesh/
sudo systemctl enable --now forge-mesh-server
# Agent (on enrolled hosts)
sudo cp deploy/systemd/forge-mesh-agent.service /etc/systemd/system/
sudo tee /etc/forge-mesh/agent.env <<EOF
FORGE_DECK_URL=http://<deck-ip>:8989
FORGE_FLEET_SECRET=<fleet-secret>
FORGE_PUBKEY=<pubkey-hex>
EOF
sudo systemctl enable --now forge-mesh-agent
```
Server working directory: `/opt/forge-mesh`. Config: `/opt/forge-mesh/data/config.json`.
---
## Testing
```bash
# Backend
make test # or: go test ./...
# Frontend
cd web && npm install
npm run test # run once
npm run test:watch # watch mode
npm run test:coverage # V8 coverage
# From repo root
make test-frontend
# End-to-end
make e2e # or: ./scripts/e2e-test.sh
```
See [`docs/TESTING.md`](docs/TESTING.md) for MSW handlers, mock WebSocket/EventSource, and how to write new tests.
---
## Environment Variables
| Variable | Scope | Description |
|----------|-------|-------------|
| `FORGE_MESH_DECK_URL` | Agent | Deck base URL |
| `FORGE_MESH_FLEET_SECRET` | Agent | Fleet bearer secret |
| `FORGE_MESH_PUBKEY` | Agent | ed25519 public key hex for signed self-update |
| `FORGE_WALLET` | Agent | Fallback mining wallet |
| `AF_TUNNEL_TOKEN` | Launcher | Cloudflare tunnel token (overrides file) |
| `AF_TUNNEL_EXTERNAL` | Launcher | Set to `1` automatically when sidecar starts |
| `AF_NO_BROWSER` | Launcher | Skip opening a browser |
| `AF_CONFIG` | Launcher | Override config path |
| `AF_DATA_DIR` | Launcher | Data directory path |
| `AF_SERVER_BIN` | Launcher | Path to `forge-mesh-server` binary |
| `AF_DECK_URL` | Launcher | Browser URL (default `http://localhost:8989`) |
---
## Known Limits
See [`docs/PROBLEMS.md`](docs/PROBLEMS.md) for the full list.
- **No worm-style LAN spread.** Enrolled-only; gated by earn-before-burn phenotype check.
- **WireGuard is operator-managed.** No auto-provisioning of keys, configs, or NAT traversal.
- **GPU tiers need drivers pre-installed.** The summon installer does not install GPU stacks.
- **Court requires an LLM you operate.** No bundled model.
- **SQLite is single-node.** Multi-deck HA and Postgres are out of scope.
- **Signature verify is dev-grade.** Dev builds may skip verification with a warning.
- **USB copies are not hardened live media.** Rotate all secrets before production use.
---
## Security Notes
- Change `auth.basic_password` and `auth.fleet_secret` before exposing the deck.
- Do not commit `data/signing.key`, `data/cloudflared-token.txt`, or `data/forge-mesh.db`.
- L0–L4 clearance gates are policy-enforced, not cryptographic. Compromised deck credentials bypass all gates.
- USB copies carry your fleet secret — treat the device like a key.
- Subnet sweeps (`/api/v1/fleet/subnets/sweep`) only probe operator-declared CIDRs.
---
## Project Layout
```
aetherforge-linux/
├── cmd/
│ ├── agent/ # forge-mesh-agent binary
│ ├── forge/ # forge CLI (build/sign artifacts)
│ └── server/ # forge-mesh-server + embedded webroot
├── internal/
│ ├── alerts/ # Telegram notifier
│ ├── api/ # HTTP server, router, handlers, WS hub
│ ├── auth/ # Basic auth, fleet secret middleware, ticket store
│ ├── config/ # Config loader
│ ├── court/ # Singular Machine Court + Seer event hub
│ ├── db/ # SQLite open/migrate
│ ├── erasure/ # Reed-Solomon erasure coding (T12)
│ ├── fleet/ # Host store, fleet hub, tier chain, recon, atlas
│ ├── forge/ # Build pipeline, key pair, artifact signing
│ ├── mining/ # Mining chain, tier implementations, mock miner
│ ├── policy/ # Mining profile policy
│ ├── stratum/ # TCP stratum proxy (XMR + RVN)
│ └── testutil/ # Shared test helpers
├── web/ # React + Vite + Tailwind Command Deck
├── deploy/systemd/ # systemd unit files
├── scripts/ # LAUNCH.sh, pack-usb.sh, test runners, install template
├── data/ # Runtime data (config, SQLite, signing key, artifacts)
├── docs/ # TESTING.md, PROBLEMS.md
├── dist/ # Packed portable tarballs
└── Makefile
```