271 lines
12 KiB
Markdown
271 lines
12 KiB
Markdown
# Autonomous AI Publishing System — Architecture
|
|
|
|
## Overview
|
|
|
|
A fully autonomous, self-hosted publishing platform that continuously discovers valuable topics, generates original useful content, and publishes to a collection of evergreen authority websites.
|
|
|
|
## Sites (Vertical Authority Domains)
|
|
|
|
| Site | Domain | CT ID | IP | Port |
|
|
|------|--------|-------|-----|------|
|
|
| AI | ai.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Tech | tech.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Science | science.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Crypto | crypto.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Linux | linux.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Gaming | gaming.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| DIY | diy.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
| Guides | guides.thetempleofdoom.com | TBD | TBD | 5000 |
|
|
|
|
## System Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ CRON ORCHESTRATOR (MacBook) │
|
|
│ Every morning @ 6AM Pacific │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ TREND DISCOVERY ENGINE │
|
|
│ Sources: News APIs, RSS, Reddit, HN, GitHub, Google Trends, │
|
|
│ arXiv, Stack Overflow, Twitter/X, seasonal calendar │
|
|
│ Output: Scored topic list with vertical assignments │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ RESEARCH AGENT │
|
|
│ Per topic: multi-source extraction → facts, stats, citations, │
|
|
│ FAQs, timelines, misconceptions, examples │
|
|
│ LLM: GamingPC ornith:latest for deep research │
|
|
│ Output: Knowledge Package (JSON) → stored in SQLite │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ MULTI-AGENT WRITING PIPELINE │
|
|
│ Outline Agent → Technical Writer → SEO Writer → Copy Editor │
|
|
│ → Fact Checker → Quality Reviewer │
|
|
│ LLMs: qwen3.5:4b (fast gate) + ornith:latest (verify) │
|
|
│ Output: Polished Markdown article + frontmatter │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ IMAGE GENERATION │
|
|
│ FLUX via FAL.ai for hero images, diagrams, charts │
|
|
│ Output: WebP images in site assets/ │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ SEO PROCESSOR │
|
|
│ Meta titles, descriptions, OG tags, JSON-LD, schema.org, │
|
|
│ canonical URLs, breadcrumbs, XML sitemap, robots.txt │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ SITE BUILDER (per vertical) │
|
|
│ Static site generator: Markdown → HTML, rebuilds on new content │
|
|
│ Updates: homepage, category pages, RSS, sitemap, related links │
|
|
│ Built-in search via pagefind │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ DEPLOYMENT │
|
|
│ 1. Build static site locally │
|
|
│ 2. scp to Proxmox CT │
|
|
│ 3. Restart nginx/service │
|
|
│ 4. Ping sitemap to Google/Bing │
|
|
│ 5. Verify live │
|
|
└──────────────────────────┬───────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ ANALYTICS + LEARNING LOOP │
|
|
│ Nightly: analyze pageviews, time-on-page, bounce rate │
|
|
│ Feed back into topic scoring → improve selection over time │
|
|
│ Plausible-style privacy-first analytics, self-hosted │
|
|
└──────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Service APIs (REST)
|
|
|
|
Every component is an independent API service:
|
|
|
|
### Trend Discovery API (port 5100)
|
|
- `POST /api/trends/discover` — Run full discovery scan
|
|
- `GET /api/trends/scored` — Get scored topic list
|
|
- `GET /api/trends/sources` — List active data sources
|
|
|
|
### Research API (port 5101)
|
|
- `POST /api/research/topic` — Research a topic, return knowledge package
|
|
- `GET /api/research/package/{id}` — Get stored knowledge package
|
|
- `GET /api/research/sources/{topic}` — Raw sources for a topic
|
|
|
|
### Writing API (port 5102)
|
|
- `POST /api/write/article` — Generate article from knowledge package
|
|
- `GET /api/write/draft/{id}` — Get draft
|
|
- `POST /api/write/review/{id}` — Request quality review
|
|
|
|
### SEO API (port 5103)
|
|
- `POST /api/seo/optimize` — SEO-optimize an article
|
|
- `POST /api/seo/sitemap` — Generate sitemap for a site
|
|
- `POST /api/seo/structured-data` — Generate JSON-LD
|
|
|
|
### Image API (port 5104)
|
|
- `POST /api/images/generate` — Generate image for article
|
|
- `POST /api/images/optimize` — Optimize existing image
|
|
|
|
### Deploy API (port 5105)
|
|
- `POST /api/deploy/site/{name}` — Build and deploy a site
|
|
- `POST /api/deploy/all` — Deploy all sites
|
|
- `GET /api/deploy/status/{name}` — Deployment status
|
|
|
|
### Admin Dashboard (port 5106)
|
|
- Full control panel UI
|
|
- Topic approval, manual triggers, analytics views
|
|
|
|
## Database Schema
|
|
|
|
### topics
|
|
```sql
|
|
CREATE TABLE topics (
|
|
id INTEGER PRIMARY KEY,
|
|
title TEXT NOT NULL,
|
|
vertical TEXT NOT NULL, -- ai, tech, science, crypto, linux, gaming, diy, guides
|
|
trend_score REAL,
|
|
search_volume INTEGER,
|
|
competition_score REAL,
|
|
freshness_score REAL,
|
|
evergreen_score REAL,
|
|
composite_score REAL,
|
|
sources TEXT, -- JSON array
|
|
status TEXT DEFAULT 'discovered', -- discovered, approved, researching, writing, reviewing, published, rejected
|
|
knowledge_package_id INTEGER,
|
|
article_id INTEGER,
|
|
published_url TEXT,
|
|
published_at TIMESTAMP,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### knowledge_packages
|
|
```sql
|
|
CREATE TABLE knowledge_packages (
|
|
id INTEGER PRIMARY KEY,
|
|
topic_id INTEGER,
|
|
facts TEXT, -- JSON
|
|
stats TEXT, -- JSON
|
|
definitions TEXT, -- JSON
|
|
faqs TEXT, -- JSON
|
|
misconceptions TEXT, -- JSON
|
|
timeline TEXT, -- JSON
|
|
citations TEXT, -- JSON
|
|
examples TEXT, -- JSON
|
|
related_concepts TEXT, -- JSON
|
|
raw_sources TEXT, -- JSON
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### articles
|
|
```sql
|
|
CREATE TABLE articles (
|
|
id INTEGER PRIMARY KEY,
|
|
topic_id INTEGER,
|
|
vertical TEXT,
|
|
title TEXT,
|
|
slug TEXT UNIQUE,
|
|
content_md TEXT,
|
|
content_html TEXT,
|
|
seo_title TEXT,
|
|
seo_description TEXT,
|
|
og_image TEXT,
|
|
json_ld TEXT,
|
|
word_count INTEGER,
|
|
reading_time_minutes INTEGER,
|
|
status TEXT DEFAULT 'draft',
|
|
published_at TIMESTAMP,
|
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### analytics
|
|
```sql
|
|
CREATE TABLE analytics (
|
|
id INTEGER PRIMARY KEY,
|
|
article_id INTEGER,
|
|
vertical TEXT,
|
|
pageviews INTEGER DEFAULT 0,
|
|
unique_visitors INTEGER DEFAULT 0,
|
|
avg_time_on_page REAL,
|
|
bounce_rate REAL,
|
|
referrers TEXT, -- JSON
|
|
recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### performance_learning
|
|
```sql
|
|
CREATE TABLE performance_learning (
|
|
id INTEGER PRIMARY KEY,
|
|
vertical TEXT,
|
|
top_performing_patterns TEXT, -- JSON
|
|
headline_formats TEXT, -- JSON
|
|
optimal_word_count INTEGER,
|
|
best_publish_times TEXT, -- JSON
|
|
keyword_insights TEXT, -- JSON
|
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
```
|
|
|
|
## LLM Strategy
|
|
|
|
| Task | Model | Host | Reasoning |
|
|
|------|-------|------|-----------|
|
|
| Topic scoring | qwen3.5:4b | MacBook | Fast, cheap |
|
|
| Research extraction | ornith:latest | GamingPC | Deep reasoning |
|
|
| Outline generation | qwen3.5:4b | MacBook | Structural, fast |
|
|
| Technical writing | ornith:latest | GamingPC | Quality matters |
|
|
| SEO optimization | qwen3.5:4b | MacBook | Pattern matching |
|
|
| Copy editing | qwen3.5:4b | MacBook | Fast iteration |
|
|
| Fact checking | ornith:latest | GamingPC | Accuracy critical |
|
|
| Quality review | ornith:latest | GamingPC | Final gate |
|
|
|
|
## Deployment Strategy
|
|
|
|
Each site is a standalone Proxmox CT running:
|
|
- Python Flask/FastAPI (static site generator)
|
|
- nginx (serving static files)
|
|
- pagefind (search index)
|
|
- cloudflared (tunnel to CF)
|
|
|
|
Core orchestrator runs on MacBook via cron (launchd-supervised).
|
|
|
|
## Security Model
|
|
|
|
- All APIs internal-only (10.30.20.0/24)
|
|
- Admin dashboard: localhost only, auth via Hermes
|
|
- Cloudflare tunnels: only expose nginx on :80
|
|
- No secrets in code — env vars only
|
|
- Gitea private repos for all sites
|
|
|
|
## Backup Strategy
|
|
|
|
- All article content in Gitea (git history = backup)
|
|
- SQLite DBs synced to iCloud daily
|
|
- Proxmox CT snapshots weekly
|
|
- Knowledge packages exported to JSON nightly
|
|
|
|
## Monitoring
|
|
|
|
- Health checks on all 8 site CTs
|
|
- Orchestrator pipeline status dashboard
|
|
- LLM usage/cost tracking
|
|
- Publish success/failure alerts via Telegram
|