# Autonomous AI Publishing System — Architecture

## Overview

A fully autonomous, self-hosted publishing platform that continuously discovers valuable topics, generates original useful content, and publishes to a collection of evergreen authority websites.

## Sites (Vertical Authority Domains)

| Site | Domain | CT ID | IP | Port |
|------|--------|-------|-----|------|
| AI | ai.thetempleofdoom.com | TBD | TBD | 5000 |
| Tech | tech.thetempleofdoom.com | TBD | TBD | 5000 |
| Science | science.thetempleofdoom.com | TBD | TBD | 5000 |
| Crypto | crypto.thetempleofdoom.com | TBD | TBD | 5000 |
| Linux | linux.thetempleofdoom.com | TBD | TBD | 5000 |
| Gaming | gaming.thetempleofdoom.com | TBD | TBD | 5000 |
| DIY | diy.thetempleofdoom.com | TBD | TBD | 5000 |
| Guides | guides.thetempleofdoom.com | TBD | TBD | 5000 |

## System Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│                    CRON ORCHESTRATOR (MacBook)                    │
│                   Every morning @ 6AM Pacific                     │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    TREND DISCOVERY ENGINE                         │
│  Sources: News APIs, RSS, Reddit, HN, GitHub, Google Trends,     │
│           arXiv, Stack Overflow, Twitter/X, seasonal calendar    │
│  Output: Scored topic list with vertical assignments             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                      RESEARCH AGENT                               │
│  Per topic: multi-source extraction → facts, stats, citations,   │
│  FAQs, timelines, misconceptions, examples                       │
│  LLM: GamingPC ornith:latest for deep research                   │
│  Output: Knowledge Package (JSON) → stored in SQLite             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                 MULTI-AGENT WRITING PIPELINE                      │
│  Outline Agent → Technical Writer → SEO Writer → Copy Editor     │
│  → Fact Checker → Quality Reviewer                               │
│  LLMs: qwen3.5:4b (fast gate) + ornith:latest (verify)           │
│  Output: Polished Markdown article + frontmatter                 │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    IMAGE GENERATION                               │
│  FLUX via FAL.ai for hero images, diagrams, charts               │
│  Output: WebP images in site assets/                             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    SEO PROCESSOR                                  │
│  Meta titles, descriptions, OG tags, JSON-LD, schema.org,        │
│  canonical URLs, breadcrumbs, XML sitemap, robots.txt            │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    SITE BUILDER (per vertical)                    │
│  Static site generator: Markdown → HTML, rebuilds on new content │
│  Updates: homepage, category pages, RSS, sitemap, related links  │
│  Built-in search via pagefind                                    │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    DEPLOYMENT                                     │
│  1. Build static site locally                                     │
│  2. scp to Proxmox CT                                            │
│  3. Restart nginx/service                                        │
│  4. Ping sitemap to Google/Bing                                  │
│  5. Verify live                                                   │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    ANALYTICS + LEARNING LOOP                      │
│  Nightly: analyze pageviews, time-on-page, bounce rate            │
│  Feed back into topic scoring → improve selection over time      │
│  Plausible-style privacy-first analytics, self-hosted            │
└──────────────────────────────────────────────────────────────────┘
```

## Service APIs (REST)

Every component is an independent API service:

### Trend Discovery API (port 5100)
- `POST /api/trends/discover` — Run full discovery scan
- `GET /api/trends/scored` — Get scored topic list
- `GET /api/trends/sources` — List active data sources

### Research API (port 5101)
- `POST /api/research/topic` — Research a topic, return knowledge package
- `GET /api/research/package/{id}` — Get stored knowledge package
- `GET /api/research/sources/{topic}` — Raw sources for a topic

### Writing API (port 5102)
- `POST /api/write/article` — Generate article from knowledge package
- `GET /api/write/draft/{id}` — Get draft
- `POST /api/write/review/{id}` — Request quality review

### SEO API (port 5103)
- `POST /api/seo/optimize` — SEO-optimize an article
- `POST /api/seo/sitemap` — Generate sitemap for a site
- `POST /api/seo/structured-data` — Generate JSON-LD

### Image API (port 5104)
- `POST /api/images/generate` — Generate image for article
- `POST /api/images/optimize` — Optimize existing image

### Deploy API (port 5105)
- `POST /api/deploy/site/{name}` — Build and deploy a site
- `POST /api/deploy/all` — Deploy all sites
- `GET /api/deploy/status/{name}` — Deployment status

### Admin Dashboard (port 5106)
- Full control panel UI
- Topic approval, manual triggers, analytics views

## Database Schema

### topics
```sql
CREATE TABLE topics (
    id INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    vertical TEXT NOT NULL,  -- ai, tech, science, crypto, linux, gaming, diy, guides
    trend_score REAL,
    search_volume INTEGER,
    competition_score REAL,
    freshness_score REAL,
    evergreen_score REAL,
    composite_score REAL,
    sources TEXT,  -- JSON array
    status TEXT DEFAULT 'discovered',  -- discovered, approved, researching, writing, reviewing, published, rejected
    knowledge_package_id INTEGER,
    article_id INTEGER,
    published_url TEXT,
    published_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### knowledge_packages
```sql
CREATE TABLE knowledge_packages (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER,
    facts TEXT,  -- JSON
    stats TEXT,  -- JSON
    definitions TEXT,  -- JSON
    faqs TEXT,  -- JSON
    misconceptions TEXT,  -- JSON
    timeline TEXT,  -- JSON
    citations TEXT,  -- JSON
    examples TEXT,  -- JSON
    related_concepts TEXT,  -- JSON
    raw_sources TEXT,  -- JSON
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### articles
```sql
CREATE TABLE articles (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER,
    vertical TEXT,
    title TEXT,
    slug TEXT UNIQUE,
    content_md TEXT,
    content_html TEXT,
    seo_title TEXT,
    seo_description TEXT,
    og_image TEXT,
    json_ld TEXT,
    word_count INTEGER,
    reading_time_minutes INTEGER,
    status TEXT DEFAULT 'draft',
    published_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### analytics
```sql
CREATE TABLE analytics (
    id INTEGER PRIMARY KEY,
    article_id INTEGER,
    vertical TEXT,
    pageviews INTEGER DEFAULT 0,
    unique_visitors INTEGER DEFAULT 0,
    avg_time_on_page REAL,
    bounce_rate REAL,
    referrers TEXT,  -- JSON
    recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

### performance_learning
```sql
CREATE TABLE performance_learning (
    id INTEGER PRIMARY KEY,
    vertical TEXT,
    top_performing_patterns TEXT,  -- JSON
    headline_formats TEXT,  -- JSON
    optimal_word_count INTEGER,
    best_publish_times TEXT,  -- JSON
    keyword_insights TEXT,  -- JSON
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```

## LLM Strategy

| Task | Model | Host | Reasoning |
|------|-------|------|-----------|
| Topic scoring | qwen3.5:4b | MacBook | Fast, cheap |
| Research extraction | ornith:latest | GamingPC | Deep reasoning |
| Outline generation | qwen3.5:4b | MacBook | Structural, fast |
| Technical writing | ornith:latest | GamingPC | Quality matters |
| SEO optimization | qwen3.5:4b | MacBook | Pattern matching |
| Copy editing | qwen3.5:4b | MacBook | Fast iteration |
| Fact checking | ornith:latest | GamingPC | Accuracy critical |
| Quality review | ornith:latest | GamingPC | Final gate |

## Deployment Strategy

Each site is a standalone Proxmox CT running:
- Python Flask/FastAPI (static site generator)
- nginx (serving static files)
- pagefind (search index)
- cloudflared (tunnel to CF)

Core orchestrator runs on MacBook via cron (launchd-supervised).

## Security Model

- All APIs internal-only (10.30.20.0/24)
- Admin dashboard: localhost only, auth via Hermes
- Cloudflare tunnels: only expose nginx on :80
- No secrets in code — env vars only
- Gitea private repos for all sites

## Backup Strategy

- All article content in Gitea (git history = backup)
- SQLite DBs synced to iCloud daily
- Proxmox CT snapshots weekly
- Knowledge packages exported to JSON nightly

## Monitoring

- Health checks on all 8 site CTs
- Orchestrator pipeline status dashboard
- LLM usage/cost tracking
- Publish success/failure alerts via Telegram