Autonomous Publishing System — full stack: orchestrator, 8 vertical sites, admin dashboard, analytics, cron pipeline

2026-08-03 21:44:26 -07:00
commit 493776b9f8
16 changed files with 2962 additions and 0 deletions
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,270 @@
+# Autonomous AI Publishing System — Architecture
+
+## Overview
+
+A fully autonomous, self-hosted publishing platform that continuously discovers valuable topics, generates original useful content, and publishes to a collection of evergreen authority websites.
+
+## Sites (Vertical Authority Domains)
+
+| Site | Domain | CT ID | IP | Port |
+|------|--------|-------|-----|------|
+| AI | ai.thetempleofdoom.com | TBD | TBD | 5000 |
+| Tech | tech.thetempleofdoom.com | TBD | TBD | 5000 |
+| Science | science.thetempleofdoom.com | TBD | TBD | 5000 |
+| Crypto | crypto.thetempleofdoom.com | TBD | TBD | 5000 |
+| Linux | linux.thetempleofdoom.com | TBD | TBD | 5000 |
+| Gaming | gaming.thetempleofdoom.com | TBD | TBD | 5000 |
+| DIY | diy.thetempleofdoom.com | TBD | TBD | 5000 |
+| Guides | guides.thetempleofdoom.com | TBD | TBD | 5000 |
+
+## System Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│                    CRON ORCHESTRATOR (MacBook)                    │
+│                   Every morning @ 6AM Pacific                     │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    TREND DISCOVERY ENGINE                         │
+│  Sources: News APIs, RSS, Reddit, HN, GitHub, Google Trends,     │
+│           arXiv, Stack Overflow, Twitter/X, seasonal calendar    │
+│  Output: Scored topic list with vertical assignments             │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                      RESEARCH AGENT                               │
+│  Per topic: multi-source extraction → facts, stats, citations,   │
+│  FAQs, timelines, misconceptions, examples                       │
+│  LLM: GamingPC ornith:latest for deep research                   │
+│  Output: Knowledge Package (JSON) → stored in SQLite             │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                 MULTI-AGENT WRITING PIPELINE                      │
+│  Outline Agent → Technical Writer → SEO Writer → Copy Editor     │
+│  → Fact Checker → Quality Reviewer                               │
+│  LLMs: qwen3.5:4b (fast gate) + ornith:latest (verify)           │
+│  Output: Polished Markdown article + frontmatter                 │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    IMAGE GENERATION                               │
+│  FLUX via FAL.ai for hero images, diagrams, charts               │
+│  Output: WebP images in site assets/                             │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    SEO PROCESSOR                                  │
+│  Meta titles, descriptions, OG tags, JSON-LD, schema.org,        │
+│  canonical URLs, breadcrumbs, XML sitemap, robots.txt            │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    SITE BUILDER (per vertical)                    │
+│  Static site generator: Markdown → HTML, rebuilds on new content │
+│  Updates: homepage, category pages, RSS, sitemap, related links  │
+│  Built-in search via pagefind                                    │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    DEPLOYMENT                                     │
+│  1. Build static site locally                                     │
+│  2. scp to Proxmox CT                                            │
+│  3. Restart nginx/service                                        │
+│  4. Ping sitemap to Google/Bing                                  │
+│  5. Verify live                                                   │
+└──────────────────────────┬───────────────────────────────────────┘
+                           │
+                           ▼
+┌──────────────────────────────────────────────────────────────────┐
+│                    ANALYTICS + LEARNING LOOP                      │
+│  Nightly: analyze pageviews, time-on-page, bounce rate            │
+│  Feed back into topic scoring → improve selection over time      │
+│  Plausible-style privacy-first analytics, self-hosted            │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+## Service APIs (REST)
+
+Every component is an independent API service:
+
+### Trend Discovery API (port 5100)
+- `POST /api/trends/discover` — Run full discovery scan
+- `GET /api/trends/scored` — Get scored topic list
+- `GET /api/trends/sources` — List active data sources
+
+### Research API (port 5101)
+- `POST /api/research/topic` — Research a topic, return knowledge package
+- `GET /api/research/package/{id}` — Get stored knowledge package
+- `GET /api/research/sources/{topic}` — Raw sources for a topic
+
+### Writing API (port 5102)
+- `POST /api/write/article` — Generate article from knowledge package
+- `GET /api/write/draft/{id}` — Get draft
+- `POST /api/write/review/{id}` — Request quality review
+
+### SEO API (port 5103)
+- `POST /api/seo/optimize` — SEO-optimize an article
+- `POST /api/seo/sitemap` — Generate sitemap for a site
+- `POST /api/seo/structured-data` — Generate JSON-LD
+
+### Image API (port 5104)
+- `POST /api/images/generate` — Generate image for article
+- `POST /api/images/optimize` — Optimize existing image
+
+### Deploy API (port 5105)
+- `POST /api/deploy/site/{name}` — Build and deploy a site
+- `POST /api/deploy/all` — Deploy all sites
+- `GET /api/deploy/status/{name}` — Deployment status
+
+### Admin Dashboard (port 5106)
+- Full control panel UI
+- Topic approval, manual triggers, analytics views
+
+## Database Schema
+
+### topics
+```sql
+CREATE TABLE topics (
+    id INTEGER PRIMARY KEY,
+    title TEXT NOT NULL,
+    vertical TEXT NOT NULL,  -- ai, tech, science, crypto, linux, gaming, diy, guides
+    trend_score REAL,
+    search_volume INTEGER,
+    competition_score REAL,
+    freshness_score REAL,
+    evergreen_score REAL,
+    composite_score REAL,
+    sources TEXT,  -- JSON array
+    status TEXT DEFAULT 'discovered',  -- discovered, approved, researching, writing, reviewing, published, rejected
+    knowledge_package_id INTEGER,
+    article_id INTEGER,
+    published_url TEXT,
+    published_at TIMESTAMP,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### knowledge_packages
+```sql
+CREATE TABLE knowledge_packages (
+    id INTEGER PRIMARY KEY,
+    topic_id INTEGER,
+    facts TEXT,  -- JSON
+    stats TEXT,  -- JSON
+    definitions TEXT,  -- JSON
+    faqs TEXT,  -- JSON
+    misconceptions TEXT,  -- JSON
+    timeline TEXT,  -- JSON
+    citations TEXT,  -- JSON
+    examples TEXT,  -- JSON
+    related_concepts TEXT,  -- JSON
+    raw_sources TEXT,  -- JSON
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### articles
+```sql
+CREATE TABLE articles (
+    id INTEGER PRIMARY KEY,
+    topic_id INTEGER,
+    vertical TEXT,
+    title TEXT,
+    slug TEXT UNIQUE,
+    content_md TEXT,
+    content_html TEXT,
+    seo_title TEXT,
+    seo_description TEXT,
+    og_image TEXT,
+    json_ld TEXT,
+    word_count INTEGER,
+    reading_time_minutes INTEGER,
+    status TEXT DEFAULT 'draft',
+    published_at TIMESTAMP,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### analytics
+```sql
+CREATE TABLE analytics (
+    id INTEGER PRIMARY KEY,
+    article_id INTEGER,
+    vertical TEXT,
+    pageviews INTEGER DEFAULT 0,
+    unique_visitors INTEGER DEFAULT 0,
+    avg_time_on_page REAL,
+    bounce_rate REAL,
+    referrers TEXT,  -- JSON
+    recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+### performance_learning
+```sql
+CREATE TABLE performance_learning (
+    id INTEGER PRIMARY KEY,
+    vertical TEXT,
+    top_performing_patterns TEXT,  -- JSON
+    headline_formats TEXT,  -- JSON
+    optimal_word_count INTEGER,
+    best_publish_times TEXT,  -- JSON
+    keyword_insights TEXT,  -- JSON
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+## LLM Strategy
+
+| Task | Model | Host | Reasoning |
+|------|-------|------|-----------|
+| Topic scoring | qwen3.5:4b | MacBook | Fast, cheap |
+| Research extraction | ornith:latest | GamingPC | Deep reasoning |
+| Outline generation | qwen3.5:4b | MacBook | Structural, fast |
+| Technical writing | ornith:latest | GamingPC | Quality matters |
+| SEO optimization | qwen3.5:4b | MacBook | Pattern matching |
+| Copy editing | qwen3.5:4b | MacBook | Fast iteration |
+| Fact checking | ornith:latest | GamingPC | Accuracy critical |
+| Quality review | ornith:latest | GamingPC | Final gate |
+
+## Deployment Strategy
+
+Each site is a standalone Proxmox CT running:
+- Python Flask/FastAPI (static site generator)
+- nginx (serving static files)
+- pagefind (search index)
+- cloudflared (tunnel to CF)
+
+Core orchestrator runs on MacBook via cron (launchd-supervised).
+
+## Security Model
+
+- All APIs internal-only (10.30.20.0/24)
+- Admin dashboard: localhost only, auth via Hermes
+- Cloudflare tunnels: only expose nginx on :80
+- No secrets in code — env vars only
+- Gitea private repos for all sites
+
+## Backup Strategy
+
+- All article content in Gitea (git history = backup)
+- SQLite DBs synced to iCloud daily
+- Proxmox CT snapshots weekly
+- Knowledge packages exported to JSON nightly
+
+## Monitoring
+
+- Health checks on all 8 site CTs
+- Orchestrator pipeline status dashboard
+- LLM usage/cost tracking
+- Publish success/failure alerts via Telegram