drjones/auto-publisher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
auto-publisher/docs/ARCHITECTURE.md

12 KiB
Raw Permalink Blame History

Autonomous AI Publishing System — Architecture

Overview

A fully autonomous, self-hosted publishing platform that continuously discovers valuable topics, generates original useful content, and publishes to a collection of evergreen authority websites.

Sites (Vertical Authority Domains)

Site Domain CT ID IP Port
AI ai.thetempleofdoom.com TBD TBD 5000
Tech tech.thetempleofdoom.com TBD TBD 5000
Science science.thetempleofdoom.com TBD TBD 5000
Crypto crypto.thetempleofdoom.com TBD TBD 5000
Linux linux.thetempleofdoom.com TBD TBD 5000
Gaming gaming.thetempleofdoom.com TBD TBD 5000
DIY diy.thetempleofdoom.com TBD TBD 5000
Guides guides.thetempleofdoom.com TBD TBD 5000

System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                    CRON ORCHESTRATOR (MacBook)                    │
│                   Every morning @ 6AM Pacific                     │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    TREND DISCOVERY ENGINE                         │
│  Sources: News APIs, RSS, Reddit, HN, GitHub, Google Trends,     │
│           arXiv, Stack Overflow, Twitter/X, seasonal calendar    │
│  Output: Scored topic list with vertical assignments             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                      RESEARCH AGENT                               │
│  Per topic: multi-source extraction → facts, stats, citations,   │
│  FAQs, timelines, misconceptions, examples                       │
│  LLM: GamingPC ornith:latest for deep research                   │
│  Output: Knowledge Package (JSON) → stored in SQLite             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                 MULTI-AGENT WRITING PIPELINE                      │
│  Outline Agent → Technical Writer → SEO Writer → Copy Editor     │
│  → Fact Checker → Quality Reviewer                               │
│  LLMs: qwen3.5:4b (fast gate) + ornith:latest (verify)           │
│  Output: Polished Markdown article + frontmatter                 │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    IMAGE GENERATION                               │
│  FLUX via FAL.ai for hero images, diagrams, charts               │
│  Output: WebP images in site assets/                             │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    SEO PROCESSOR                                  │
│  Meta titles, descriptions, OG tags, JSON-LD, schema.org,        │
│  canonical URLs, breadcrumbs, XML sitemap, robots.txt            │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    SITE BUILDER (per vertical)                    │
│  Static site generator: Markdown → HTML, rebuilds on new content │
│  Updates: homepage, category pages, RSS, sitemap, related links  │
│  Built-in search via pagefind                                    │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    DEPLOYMENT                                     │
│  1. Build static site locally                                     │
│  2. scp to Proxmox CT                                            │
│  3. Restart nginx/service                                        │
│  4. Ping sitemap to Google/Bing                                  │
│  5. Verify live                                                   │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│                    ANALYTICS + LEARNING LOOP                      │
│  Nightly: analyze pageviews, time-on-page, bounce rate            │
│  Feed back into topic scoring → improve selection over time      │
│  Plausible-style privacy-first analytics, self-hosted            │
└──────────────────────────────────────────────────────────────────┘

Service APIs (REST)

Every component is an independent API service:

Trend Discovery API (port 5100)

  • POST /api/trends/discover — Run full discovery scan
  • GET /api/trends/scored — Get scored topic list
  • GET /api/trends/sources — List active data sources

Research API (port 5101)

  • POST /api/research/topic — Research a topic, return knowledge package
  • GET /api/research/package/{id} — Get stored knowledge package
  • GET /api/research/sources/{topic} — Raw sources for a topic

Writing API (port 5102)

  • POST /api/write/article — Generate article from knowledge package
  • GET /api/write/draft/{id} — Get draft
  • POST /api/write/review/{id} — Request quality review

SEO API (port 5103)

  • POST /api/seo/optimize — SEO-optimize an article
  • POST /api/seo/sitemap — Generate sitemap for a site
  • POST /api/seo/structured-data — Generate JSON-LD

Image API (port 5104)

  • POST /api/images/generate — Generate image for article
  • POST /api/images/optimize — Optimize existing image

Deploy API (port 5105)

  • POST /api/deploy/site/{name} — Build and deploy a site
  • POST /api/deploy/all — Deploy all sites
  • GET /api/deploy/status/{name} — Deployment status

Admin Dashboard (port 5106)

  • Full control panel UI
  • Topic approval, manual triggers, analytics views

Database Schema

topics

CREATE TABLE topics (
    id INTEGER PRIMARY KEY,
    title TEXT NOT NULL,
    vertical TEXT NOT NULL,  -- ai, tech, science, crypto, linux, gaming, diy, guides
    trend_score REAL,
    search_volume INTEGER,
    competition_score REAL,
    freshness_score REAL,
    evergreen_score REAL,
    composite_score REAL,
    sources TEXT,  -- JSON array
    status TEXT DEFAULT 'discovered',  -- discovered, approved, researching, writing, reviewing, published, rejected
    knowledge_package_id INTEGER,
    article_id INTEGER,
    published_url TEXT,
    published_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

knowledge_packages

CREATE TABLE knowledge_packages (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER,
    facts TEXT,  -- JSON
    stats TEXT,  -- JSON
    definitions TEXT,  -- JSON
    faqs TEXT,  -- JSON
    misconceptions TEXT,  -- JSON
    timeline TEXT,  -- JSON
    citations TEXT,  -- JSON
    examples TEXT,  -- JSON
    related_concepts TEXT,  -- JSON
    raw_sources TEXT,  -- JSON
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

articles

CREATE TABLE articles (
    id INTEGER PRIMARY KEY,
    topic_id INTEGER,
    vertical TEXT,
    title TEXT,
    slug TEXT UNIQUE,
    content_md TEXT,
    content_html TEXT,
    seo_title TEXT,
    seo_description TEXT,
    og_image TEXT,
    json_ld TEXT,
    word_count INTEGER,
    reading_time_minutes INTEGER,
    status TEXT DEFAULT 'draft',
    published_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

analytics

CREATE TABLE analytics (
    id INTEGER PRIMARY KEY,
    article_id INTEGER,
    vertical TEXT,
    pageviews INTEGER DEFAULT 0,
    unique_visitors INTEGER DEFAULT 0,
    avg_time_on_page REAL,
    bounce_rate REAL,
    referrers TEXT,  -- JSON
    recorded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

performance_learning

CREATE TABLE performance_learning (
    id INTEGER PRIMARY KEY,
    vertical TEXT,
    top_performing_patterns TEXT,  -- JSON
    headline_formats TEXT,  -- JSON
    optimal_word_count INTEGER,
    best_publish_times TEXT,  -- JSON
    keyword_insights TEXT,  -- JSON
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

LLM Strategy

Task Model Host Reasoning
Topic scoring qwen3.5:4b MacBook Fast, cheap
Research extraction ornith:latest GamingPC Deep reasoning
Outline generation qwen3.5:4b MacBook Structural, fast
Technical writing ornith:latest GamingPC Quality matters
SEO optimization qwen3.5:4b MacBook Pattern matching
Copy editing qwen3.5:4b MacBook Fast iteration
Fact checking ornith:latest GamingPC Accuracy critical
Quality review ornith:latest GamingPC Final gate

Deployment Strategy

Each site is a standalone Proxmox CT running:

  • Python Flask/FastAPI (static site generator)
  • nginx (serving static files)
  • pagefind (search index)
  • cloudflared (tunnel to CF)

Core orchestrator runs on MacBook via cron (launchd-supervised).

Security Model

  • All APIs internal-only (10.30.20.0/24)
  • Admin dashboard: localhost only, auth via Hermes
  • Cloudflare tunnels: only expose nginx on :80
  • No secrets in code — env vars only
  • Gitea private repos for all sites

Backup Strategy

  • All article content in Gitea (git history = backup)
  • SQLite DBs synced to iCloud daily
  • Proxmox CT snapshots weekly
  • Knowledge packages exported to JSON nightly

Monitoring

  • Health checks on all 8 site CTs
  • Orchestrator pipeline status dashboard
  • LLM usage/cost tracking
  • Publish success/failure alerts via Telegram
Reference in New Issue View Git Blame Copy Permalink