From a24ab4ea792090b5b550db5bbbbe5c1321d309a9 Mon Sep 17 00:00:00 2001 From: drjones Date: Fri, 2 Oct 2026 00:32:22 +0000 Subject: [PATCH] DRACO deployed build: RAG oracle, 1855 books, 116k passages, MCP+BTCPay --- .gitignore | 5 + README.md | 182 ++++++++++++++++++ app.py | 351 +++++++++++++++++++++++++++++++++++ build_index.py | 77 ++++++++ config.example.json | 23 +++ deploy/draco.service | 14 ++ deploy/nginx-draco.conf | 17 ++ draco_core.py | 400 ++++++++++++++++++++++++++++++++++++++++ index.log | 5 + ingest.log | 79 ++++++++ ingest.py | 125 +++++++++++++ mcp_client.py | 86 +++++++++ pages.py | 170 +++++++++++++++++ pages2.py | 142 ++++++++++++++ requirements.txt | 8 + 15 files changed, 1684 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 app.py create mode 100644 build_index.py create mode 100644 config.example.json create mode 100644 deploy/draco.service create mode 100644 deploy/nginx-draco.conf create mode 100644 draco_core.py create mode 100644 index.log create mode 100644 ingest.log create mode 100644 ingest.py create mode 100644 mcp_client.py create mode 100644 pages.py create mode 100644 pages2.py create mode 100644 requirements.txt diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7d51a0b --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +config.json +draco.db +venv/ +__pycache__/ +*.pyc diff --git a/README.md b/README.md new file mode 100644 index 0000000..48d3065 --- /dev/null +++ b/README.md @@ -0,0 +1,182 @@ +# DRACO — The Book-Forged Code Oracle + +> The smartest coding AI you can **prove**. DRACO answers coding, Linux, systems, and +> security questions using retrieval (BM25 RAG) over **hundreds of real technical books** +> — the coding canon and the hacker's shelf — and cites the exact book behind every +> answer. Talk to it in the browser, call the REST API, or summon it as an **MCP tool** +> from Claude, GPT, or any agent. + +- **Live:** https://draco.thetempleofdoom.com +- **Repo:** http://10.30.20.149:3000/drjones/draco +- **Host:** Proxmox CT 174 `draco` @ `10.30.20.12` (Debian 13, 4GB RAM / 4 cores) +- **LLM:** `ornith-1.5:9b-64k` on bare-metal Ollama @ `10.30.20.29` (nightmare, 4090S) — abliterated/heretic lineage, zero-spill resident +- **Stack:** Flask + BM25 (rank_bm25 + Snowball stem) + SSE streaming + BTCPay + MCP streamable-http + +--- + +## Why "book-forged" + +Most assistants guess. DRACO retrieves. Every question is embedded-scanned against the +full library (BM25 over stemmed tokens), the top passages are stuffed into the prompt, +and the model must cite them `[n]`. The response includes a `sources` array — +**check the receipts yourself**. That's the proof behind the homepage brag. + +## Architecture + +``` + ┌────────────────────── CT 174 "draco" (10.30.20.12) ─────────────────────┐ + │ nginx :80 ──► gunicorn :8012 (app.py + FastMCP mounted at /mcp) │ +browser ── CF tunnel│ │ │ +agent ──► /mcp ────┤ /api/chat (SSE) /api/ask /api/search /api/signup /webhook/btcpay │ + │ │ │ + │ draco_core: BM25 index (~N chunks) │ sqlite: users/credits/payments│ + │ │ │ │ + │ /opt/books: original pdf/chm/djvu + text/ + manifest.json + index/ │ + └────────────────────────────────────────┼────────────────────────────────┘ + ▼ + Ollama ornith-1.5:9b-64k @ 10.30.20.29:11434 (bare metal) +``` + +## The library + +| Collection | Contents | Source | +|---|---|---| +| `library_linux` | The coding canon: C/C++, Python, Perl, PHP, Java, assembly, SQL, web, Unix/Linux internals, networks & security, math, Apache | `/Volumes/sanD/library linux` | +| `hackerpack` | Hacker Pro Pack: exploitation, malware, network attacks, crypto, wireless, scene classics | `/Volumes/sanD/dw stuff 3/hacking/HACKER PRO PACK books` | + +**Conversion pipeline** (`ingest.py`, runs on the CT): + +``` +pdf → pdftotext -layout (poppler-utils) +chm → extract_chmLib → walk html/txt → strip tags → unescape +djvu → djvutxt (djvulibre-bin) +txt → passthrough +then: null-strip, whitespace collapse, <200 chars = scanned junk → dropped +→ /opt/books/text//_.txt + manifest.json (id/title/category/chars/…) +``` + +**Index** (`build_index.py`): 300-token chunks, 60 overlap → Snowball-stem → `BM25Okapi` +→ `index/index.pkl` + `chunks.json`. Rebuild = re-run both scripts; app picks up on restart. + +--- + +## Web UI + +| Route | What | +|---|---| +| `/` | Hero (the brag, with live library stats) + **chat** — 10 free questions/day/IP, streaming, sources bar under every answer | +| `/library` | The whole hoard, filterable, every book **downloadable** | +| `/pricing` | Credit packs, Bitcoin-only | +| `/api` | Key signup + copy-paste curl examples | +| `/health` | `{"status":"ok","llm":true,"model":...,"books":...,"chunks":...}` | + +## REST API + +Auth: `X-API-Key` header (or `api_key` JSON field). Free tier = 30 credits/month (email signup). Credits never expire. + +```bash +# get a key +curl -s -X POST https://draco.thetempleofdoom.com/api/signup \ + -d '{"email":"you@domain.tld"}' +# → {"api_key":"sk-draco-..."} + +# ask (RAG + citations) — 1 credit +curl -s -X POST https://draco.thetempleofdoom.com/api/ask \ + -H 'X-API-Key: sk-draco-...' -H 'Content-Type: application/json' \ + -d '{"q":"explain the ret2libc technique with a minimal poc"}' +# → {"answer":"...[1]...[2]","sources":[{"title":"...","category":"hackerpack",...}]} + +# raw BM25 passage search — 1 credit +curl -s "https://draco.thetempleofdoom.com/api/search?q=tcp%20syn%20flood&k=5" \ + -H 'X-API-Key: sk-draco-...' + +# usage / credits +curl -s https://draco.thetempleofdoom.com/api/my-usage -H 'X-API-Key: sk-draco-...' + +# buy credits (Bitcoin via BTCPay) → checkout_url +curl -s -X POST https://draco.thetempleofdoom.com/api/create-invoice \ + -H 'X-API-Key: sk-draco-...' -d '{"plan":"crate"}' +``` + +| Code | Meaning | +|---|---| +| 200 | answer + sources | +| 401 | missing/invalid key | +| 402 | out of credits (free tier exhausted) | +| 429 | web anon limit (10/day) hit | +| 503 | Ollama unreachable | + +## MCP — summon as a tool call + +**Remote (any MCP client, zero install):** + +```json +{ "mcpServers": { "draco": { "url": "https://draco.thetempleofdoom.com/mcp" } } } +``` + +Tools: `draco_ask(question)` → cited answer · `draco_search(query, k)` → raw passages · +`draco_status()` → health + library stats. + +Streamable-http at `/mcp`. Smoke test by hand: + +```bash +curl -s https://draco.thetempleofdoom.com/mcp \ + -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \ + -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}' +``` + +**Local stdio client for your own machine** — `mcp_client.py` in this repo (talks to the +remote HTTP MCP; usable from Claude Desktop via `mcp-remote` or any stdio bridge). + +Machine-readable discovery: `/llms.txt`, `/llms-full.txt`, `/openapi.json`, +`/.well-known/ai-plugin.json`, `/.well-known/mcp-server.json`, `robots.txt` (all AI crawlers allowed), `/sitemap.xml`. + +## Pricing (Bitcoin only, BTCPay, no KYC) + +| Tier | Price | Gets | +|---|---|---| +| Web anon | free | 10 questions/day in browser | +| Free API key | free | 30 credits/month | +| **satchel** | $3 | 60 credits | +| **crate** | $10 | 250 credits | +| **hoard** | $25 | 750 credits | + +1 credit = 1 ask or 1 search. Credits never expire. Invoice webhook (`/webhook/btcpay?wh=`) +credits the key on `InvoiceSettled`. Books are always free to download — we sell answers, not files. + +## Deploy runbook (CT 174) + +```bash +# code push — the golden path +cd ~/draco && tar czf /tmp/d.tar.gz *.py config.json requirements.txt +scp /tmp/d.tar.gz root@10.30.20.85:/tmp/ +ssh root@10.30.20.85 "pct push 174 /tmp/d.tar.gz /tmp/d.tar.gz" +ssh root@10.30.20.85 "pct exec 174 -- bash -c 'cd /opt/draco && tar xzf /tmp/d.tar.gz && systemctl restart draco'" +curl -s https://draco.thetempleofdoom.com/health + +# rebuild the corpus (books → text → index) on the CT +ssh root@10.30.20.85 "pct exec 174 -- systemd-run --unit=draco-ingest \ + bash -c 'cd /opt/draco && ./venv/bin/python3 ingest.py && ./venv/bin/python3 build_index.py && systemctl restart draco'" +``` + +Layout on CT: `/opt/draco` (code, venv, draco.db) · `/opt/books` (books + text/ + index/) · +services `draco.service` + `nginx` · gunicorn binds 127.0.0.1:8012. + +## Verify everything (the checklist that was actually run) + +```bash +curl -s https://draco.thetempleofdoom.com/health # llm:true, books>0 +curl -s https://draco.thetempleofdoom.com/llms.txt # starts with "# DRACO" +curl -s https://draco.thetempleofdoom.com/robots.txt # GPTBot + Allow: / +curl -s https://draco.thetempleofdoom.com/api/library | jq .count +curl -s https://draco.thetempleofdoom.com/api/search?q=nmap -H 'X-API-Key: ' # hits +# MCP handshake + tools/call against /mcp +``` + +## Notes + +- Security content is for **education and authorized testing only**. +- `config.json` carries runtime knobs (model, k, limits, plans, BTCPay) — admin key + + BTCPay store creds live there on the CT, **never** in the repo (config.json is + git-ignored; a sanitized `config.example.json` is tracked). +- Books retain their original copyrights; the library is shared for personal use. diff --git a/app.py b/app.py new file mode 100644 index 0000000..3eba415 --- /dev/null +++ b/app.py @@ -0,0 +1,351 @@ +#!/usr/bin/env python3 +"""DRACO — The Book-Forged Code Oracle. +Flask app: home (hero + brag), library, pricing, API, MCP, SEO kit, RAG chat (SSE). +""" +import json, os, re +import requests as http +from flask import Flask, Response, jsonify, request, render_template_string, send_from_directory + +import draco_core as core +from draco_core import CFG +from pages import PAGE_HOME, BASE_CSS, NAV +from pages2 import PAGE_LIBRARY, PAGE_PRICING, PAGE_API + +app = Flask(__name__) + +# ------------------------------------------------------------------ pages +def render(page, active): + stats = core.get_stats() + nav = NAV.replace("__ACTIVE__-" + active, "on") + nav = re.sub(r"__ACTIVE__-\w+", "", nav) + head = f"" + page = page.replace("__STATS__", f"{stats['books']} books · {stats['chunks']:,} indexed passages · {stats['chars']/1e6:.0f}M chars") + page = page.replace("__BASE__", CFG["base_url"]) + ctx = dict(books=f"{stats['books']:,}", chunks=f"{stats['chunks']:,}", + mb=stats['chars'] // 1_000_000, free_day=CFG["anon_daily"], + free_monthly=CFG["free_monthly"], plans=CFG["plans"], base=CFG["base_url"]) + return render_template_string(head + page, **ctx) + +@app.route("/") +def home(): + return render(PAGE_HOME, "home") + +@app.route("/library") +def library(): + return render(PAGE_LIBRARY, "library") + +@app.route("/api/library") +def api_library(): + try: + manifest = json.load(open(os.path.join(core.BOOKS_DIR, "manifest.json"))) + except Exception: + manifest = [] + books = [{"id": m["id"], "title": m["title"][:90], "category": m["category"], + "format": m["format"], "mb": round(m["bytes"] / 1e6, 1), + "download": f"/download/{m['id']}"} + for m in manifest if m.get("txt")] + books.sort(key=lambda b: b["title"].lower()) + return jsonify(count=len(books), books=books) + +@app.route("/download/") +def download(book_id): + try: + manifest = json.load(open(os.path.join(core.BOOKS_DIR, "manifest.json"))) + except Exception: + return jsonify(error="library offline"), 503 + m = next((x for x in manifest if x["id"] == book_id), None) + if not m: + return jsonify(error="unknown book"), 404 + return send_from_directory(core.BOOKS_DIR, m["source"], as_attachment=True) + +@app.route("/pricing") +def pricing(): + return render(PAGE_PRICING, "pricing") + +@app.route("/api") +def api_page(): + return render(PAGE_API, "api") + +@app.route("/health") +def health(): + try: + r = http.post(f"{CFG['ollama_url']}/api/generate", stream=True, timeout=4, + json={"model": CFG["model"], "prompt": "ping", "stream": True, "options": {"num_predict": 1}}) + llm = r.status_code == 200 + except Exception: + llm = False + return jsonify(status="ok", llm=llm, model=CFG["model"], **core.get_stats()) + +# ------------------------------------------------------------------ chat (web, SSE) +@app.route("/api/chat", methods=["POST"]) +def chat(): + data = request.json or {} + q = (data.get("q") or "").strip() + if not q: + return jsonify(error="Empty question."), 400 + key = data.get("api_key") + user = None + if key: + user, err = core.auth_user(request) + if err: + return jsonify(error=err), 401 + ok, msg = core.consume(user, "chat", q[:80], core.client_ip(request)) + if not ok: + return jsonify(error=msg), 402 + else: + ip = core.client_ip(request) + if not core.anon_allowed(ip): + return jsonify(error=f"Free web limit reached ({CFG['anon_daily']}/day). " + f"Sign up free at {CFG['base_url']}/api for {CFG['free_monthly']}/mo API credits."), 429 + + prompt, hits = core.build_prompt(q, k=CFG.get("rag_k", 6)) + sources = [{"n": i + 1, "title": h["title"], "category": h["category"]} + for i, h in enumerate(hits)] + + def generate(): + yield f"data: {json.dumps({'type': 'sources', 'sources': sources})}\n\n" + try: + for piece in core.stream_ollama(prompt): + yield f"data: {json.dumps({'type': 'token', 'text': piece})}\n\n" + except Exception as e: + yield f"data: {json.dumps({'type': 'error', 'text': f'Model offline: {e}'})}\n\n" + yield f"data: {json.dumps({'type': 'done'})}\n\n" + + return Response(generate(), mimetype="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}) + +@app.route("/api/remaining") +def remaining(): + ip = core.client_ip(request) + return jsonify(remaining=core.anon_remaining(ip), daily=CFG["anon_daily"]) + +# ------------------------------------------------------------------ REST API (metered) +@app.route("/api/ask", methods=["POST"]) +def api_ask(): + user, err = core.auth_user(request) + if err: + return jsonify(error=err), 401 + q = ((request.json or {}).get("q") or "").strip() + if not q: + return jsonify(error="q required"), 400 + ok, msg = core.consume(user, "ask", q[:80], core.client_ip(request)) + if not ok: + return jsonify(error=msg), 402 + prompt, hits = core.build_prompt(q, k=CFG.get("rag_k", 6)) + try: + answer = core.ask_ollama(prompt) + except Exception as e: + return jsonify(error=f"Model offline: {e}"), 503 + return jsonify(question=q, answer=answer, + sources=[{"title": h["title"], "category": h["category"], + "book_id": h["book_id"], "chunk": h["chunk"]} for h in hits]) + +@app.route("/api/search", methods=["GET", "POST"]) +def api_search(): + user, err = core.auth_user(request) + if err: + return jsonify(error=err), 401 + q = (request.json or {}).get("q") if request.is_json else request.args.get("q") + q = (q or "").strip() + if not q: + return jsonify(error="q required"), 400 + ok, msg = core.consume(user, "search", q[:80], core.client_ip(request)) + if not ok: + return jsonify(error=msg), 402 + hits = core.search_library(q, k=int((request.json or {}).get("k", 8) if request.is_json else request.args.get("k", 8))) + for h in hits: + h.pop("text", None) + return jsonify(query=q, results=hits) + +@app.route("/api/my-usage", methods=["GET", "POST"]) +def my_usage(): + user, err = core.auth_user(request) + if err: + return jsonify(error=err), 401 + c = core.db() + u = c.execute("SELECT credits, free_used, total_calls FROM users WHERE id=?", (user["id"],)).fetchone() + if u["total_calls"] is None: + c.execute("UPDATE users SET total_calls=(SELECT COUNT(*) FROM usage_log WHERE user_id=?) WHERE id=?", (user["id"], user["id"])) + c.commit() + u = c.execute("SELECT credits, free_used, total_calls FROM users WHERE id=?", (user["id"],)).fetchone() + recent = [dict(r) for r in c.execute( + "SELECT kind, detail, created_at FROM usage_log WHERE user_id=? ORDER BY id DESC LIMIT 10", (user["id"],))] + c.close() + return jsonify(email=user["email"], credits=u["credits"], free_used_this_month=u["free_used"], + free_monthly=CFG["free_monthly"], total_calls=u["total_calls"] or 0, recent=recent) + +@app.route("/api/signup", methods=["POST"]) +def api_signup(): + email = (request.json or {}).get("email", "") + key, status = core.signup(email) + if not key: + return jsonify(error=status), 400 + return jsonify(email=email.strip().lower(), api_key=key, status=status, + free_monthly=CFG["free_monthly"], + note="Free tier active. Keep this key safe — it is shown once per signup.") + +@app.route("/api/create-invoice", methods=["POST"]) +def api_invoice(): + user, err = core.auth_user(request) + if err: + return jsonify(error=err), 401 + plan = (request.json or {}).get("plan", "") + inv, err = core.create_invoice(user, plan) + if err: + return jsonify(error=err), 400 + return jsonify(invoice_id=inv["id"], amount=inv.get("amount"), + checkout_url=inv.get("checkoutLink") or (inv.get("checkout") or {}).get("link")) + +# ------------------------------------------------------------------ BTCPay webhook +@app.route("/webhook/btcpay", methods=["POST"]) +def webhook(): + body = request.get_json(silent=True) or {} + # shared-secret check via webhook URL suffix + ok = request.args.get("wh") == CFG["btcpay"].get("webhook_secret") + core.handle_webhook(body, ok) + return jsonify(status="ok") + +# ------------------------------------------------------------------ SEO / agent kit +@app.route("/robots.txt") +def robots(): + ais = ("GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-Web anthropic-ai PerplexityBot " + "Perplexity-User Google-Extended GoogleOther Amazonbot Applebot-Extended Bytespider " + "cohere-ai Meta-ExternalAgent Diffbot CCBot").split() + lines = [f"User-agent: {a}\nAllow: /" for a in ais] + lines.append("User-agent: *\nAllow: /\nDisallow: /webhook/") + lines.append(f"Sitemap: {CFG['base_url']}/sitemap.xml") + return Response("\n\n".join(lines), mimetype="text/plain") + +@app.route("/llms.txt") +def llms_txt(): + return Response(render_template_string(core.load_llms_txt()), mimetype="text/plain") + +@app.route("/llms-full.txt") +def llms_full(): + return Response(render_template_string(core.load_llms_txt() + "\n\n" + core.load_llms_full()), mimetype="text/plain") + +@app.route("/.well-known/ai-plugin.json") +def ai_plugin(): + return Response(json.dumps({ + "schema_version": "v1", + "name_for_human": "DRACO — Book-Forged Code Oracle", + "name_for_model": "draco", + "description_for_human": "Coding & security answers grounded in a library of hundreds of real programming and hacking books, with citations.", + "description_for_model": "Ask coding, systems, and security questions. Answers are RAG-grounded in DRACO's book library and include citations [n] plus a sources array with titles. Use /api/ask for full answers, /api/search to find book passages. Auth: X-API-Key header, free tier available.", + "auth": {"type": "none"}, + "api": {"type": "openapi", "url": f"{CFG['base_url']}/openapi.json"}, + "logo_url": f"{CFG['base_url']}/static/logo.svg", + "contact_email": "drjones@thetempleofdoom.com", + "legal_info_url": f"{CFG['base_url']}/api" + }, indent=1), mimetype="application/json") + +@app.route("/.well-known/mcp-server.json") +def mcp_manifest(): + return Response(json.dumps({ + "$schema": "https://cdn.jsdelivr.net/npm/@modelcontextprotocol/sdk@latest/schema.json", + "name": "com.thetempleofdoom.draco/ask", + "description": "Coding & security oracle grounded in hundreds of real books. Ask, search, cite.", + "homepage": CFG["base_url"], + "remotes": [{"type": "streamable-http", "url": f"{CFG['base_url']}/mcp"}] + }, indent=1), mimetype="application/json") + +@app.route("/openapi.json") +def openapi(): + return Response(json.dumps({ + "openapi": "3.0.0", + "info": {"title": "DRACO API", "version": "1.0", + "description": "RAG coding/security oracle over hundreds of real books. Free tier: signup for API key."}, + "servers": [{"url": CFG["base_url"]}], + "paths": { + "/api/ask": {"post": {"summary": "Ask a coding/security question (RAG + citations)", + "requestBody": {"content": {"application/json": {"schema": {"type": "object", + "properties": {"q": {"type": "string"}, "api_key": {"type": "string"}}, + "required": ["q"]}}}}, + "responses": {"200": {"description": "answer + sources"}}}}, + "/api/search": {"get": {"summary": "BM25 passage search across the library", + "parameters": [{"name": "q", "in": "query", "required": True, "schema": {"type": "string"}}, + {"name": "k", "in": "query", "schema": {"type": "integer", "default": 8}}], + "responses": {"200": {"description": "passage results"}}}}, + "/api/signup": {"post": {"summary": "Get a free API key", + "requestBody": {"content": {"application/json": {"schema": {"type": "object", + "properties": {"email": {"type": "string"}}, "required": ["email"]}}}}, + "responses": {"200": {"description": "api_key"}}}}, + "/api/my-usage": {"get": {"summary": "Credits + recent calls", "responses": {"200": {"description": "usage"}}}}, + "/api/create-invoice": {"post": {"summary": "Buy credits (Bitcoin via BTCPay)", + "requestBody": {"content": {"application/json": {"schema": {"type": "object", + "properties": {"plan": {"type": "string", "enum": list(CFG["plans"].keys())}}}}}}, + "responses": {"200": {"description": "checkout_url"}}}} + } + }, indent=1), mimetype="application/json") + +@app.route("/sitemap.xml") +def sitemap(): + urls = [CFG["base_url"], f"{CFG['base_url']}/library", f"{CFG['base_url']}/pricing", + f"{CFG['base_url']}/api", f"{CFG['base_url']}/llms.txt"] + body = '\n' + for u in urls: + body += f"{u}" + body += "" + return Response(body, mimetype="application/xml") + +# ------------------------------------------------------------------ MCP (streamable-http) +try: + from mcp.server.fastmcp import FastMCP + mcp = FastMCP("draco", host="127.0.0.1", port=CFG.get("mcp_port", 8012), + streamable_http_path="/mcp") + + @mcp.tool() + def draco_ask(question: str) -> str: + """Ask DRACO a coding or security question. Answers are grounded in hundreds of + real programming and hacking books, with [n] citations and source titles.""" + prompt, hits = core.build_prompt(question, k=CFG.get("rag_k", 6)) + try: + answer = core.ask_ollama(prompt) + except Exception as e: + return f"Model offline: {e}" + src = "\n".join(f"[{i+1}] {h['title']} ({h['category']})" for i, h in enumerate(hits)) + return f"{answer}\n\nSOURCES:\n{src}" + + @mcp.tool() + def draco_search(query: str, k: int = 8) -> str: + """Search DRACO's book library (BM25) for passages matching a topic.""" + hits = core.search_library(query, k=k) + if not hits: + return "No passages matched." + return "\n\n".join(f"[{h['title']} · {h['category']} · score {h['score']}]\n{h['text'][:600]}" for h in hits) + + @mcp.tool() + def draco_status() -> str: + """DRACO health + library size.""" + s = core.get_stats() + return (f"DRACO {CFG['model']} · {s['books']} books · {s['chunks']} passages · " + f"API: {CFG['base_url']}/api · MCP: {CFG['base_url']}/mcp") + + from starlette.applications import Starlette + starlette_app = mcp.streamable_http_app() + app.mount("/mcp", starlette_app) + MCP_OK = True +except Exception as _e: # MCP optional at runtime + MCP_OK = False + MCP_ERR = str(_e) + +@app.route("/mcp-info") +def mcp_info(): + return jsonify(mcp=MCP_OK, error=MCP_ERR if not MCP_OK else None, + url=f"{CFG['base_url']}/mcp") + +# ------------------------------------------------------------------ static +@app.route("/static/logo.svg") +def logo(): + svg = ('' + '' + '' + 'D') + return Response(svg.replace("%23", "#"), mimetype="image/svg+xml") + +# ------------------------------------------------------------------ main +if __name__ == "__main__": + print(f"DRACO starting · MCP={'ok' if MCP_OK else 'FAILED: ' + MCP_ERR}") + app.run(host="127.0.0.1", port=CFG["port"], threaded=True) diff --git a/build_index.py b/build_index.py new file mode 100644 index 0000000..f49d0cd --- /dev/null +++ b/build_index.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""draco index build — chunk all extracted book text and build a BM25 index. +Run on the CT after ingest.py: python3 build_index.py [--books /opt/books] +Output: /opt/books/index/chunks.json + index.pkl (book_count.json = stats) +""" +import argparse, json, os, pickle, re, time + +BOOKS_DIR = "/opt/books" +INDEX_DIR = os.path.join(BOOKS_DIR, "index") +CHUNK_TOKENS, CHUNK_OVERLAP = 300, 60 + +def chunk_text(text, size=CHUNK_TOKENS, overlap=CHUNK_OVERLAP): + words = text.split() + if len(words) <= size: + return [" ".join(words)] if words else [] + out = [] + step = size - overlap + for i in range(0, len(words), step): + c = words[i:i + size] + if len(c) >= 40: + out.append(" ".join(c)) + if i + size >= len(words): + break + return out + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--books", default=BOOKS_DIR) + a = ap.parse_args() + books_dir = a.books + index_dir = os.path.join(books_dir, "index") + os.makedirs(index_dir, exist_ok=True) + + from rank_bm25 import BM25Okapi + from nltk.stem import SnowballStemmer + + t0 = time.time() + manifest = json.load(open(os.path.join(books_dir, "manifest.json"))) + usable = [m for m in manifest if m.get("txt")] + print(f"index: {len(usable)} usable books from manifest", flush=True) + + stemmer = SnowballStemmer("english") + stop = set("""a an and are as at be by for from has have he her his how i in is it its of on or she that the this to was were what when where which who will with you your our we they them not no but can could should would may might must do does did done if then than into over under about after before between during through against without within""".split()) + + def toks(s): + return [stemmer.stem(w) for w in re.findall(r"[a-z0-9_#+.-]{2,}", s.lower()) if w not in stop] + + chunk_records, texts = [], [] + CAP = 400 # max chunks indexed per book (≈480 pages) — keeps the fit inside 8GB + for m in usable: + try: + raw = open(os.path.join(books_dir, "text", m["txt"]), encoding="utf-8", errors="replace").read() + except OSError: + continue + for j, ch in enumerate(chunk_text(raw)[:CAP]): + chunk_records.append({"book": m["id"], "chunk": j}) + texts.append(ch) + print(f"index: {len(texts)} chunks from {len(usable)} books", flush=True) + + print("index: tokenizing...", flush=True) + corpus = [toks(t) for t in texts] + print("index: fitting BM25...", flush=True) + bm25 = BM25Okapi(corpus) + + with open(os.path.join(index_dir, "chunks.json"), "w") as f: + json.dump(chunk_records, f) + with open(os.path.join(index_dir, "index.pkl"), "wb") as f: + pickle.dump({"bm25": bm25, "texts": texts}, f, protocol=4) + + stats = {"books": len(usable), "chunks": len(texts), + "chars": sum(m["chars"] for m in usable), "built": time.strftime("%Y-%m-%dT%H:%M:%S")} + with open(os.path.join(books_dir, "book_count.json"), "w") as f: + json.dump(stats, f) + print(f"index DONE: {stats} in {time.time()-t0:.0f}s", flush=True) + +if __name__ == "__main__": + main() diff --git a/config.example.json b/config.example.json new file mode 100644 index 0000000..7ece1f2 --- /dev/null +++ b/config.example.json @@ -0,0 +1,23 @@ +{ + "model": "ornith-1.5:9b-64k", + "ollama_url": "http://10.30.20.29:11434", + "num_ctx": 8192, + "rag_k": 6, + "port": 8012, + "mcp_port": 8012, + "base_url": "https://draco.thetempleofdoom.com", + "anon_daily": 10, + "free_monthly": 30, + "admin_key": "sk-draco-admin-CHANGE_ME", + "plans": { + "satchel": {"usd": 3, "credits": 60}, + "crate": {"usd": 10, "credits": 250}, + "hoard": {"usd": 25, "credits": 750} + }, + "btcpay": { + "url": "https://btcpay.thetempleofdoom.com", + "store_id": "SET_ME", + "api_key": "SET_ME", + "webhook_secret": "SET_ME" + } +} diff --git a/deploy/draco.service b/deploy/draco.service new file mode 100644 index 0000000..2d06128 --- /dev/null +++ b/deploy/draco.service @@ -0,0 +1,14 @@ +[Unit] +Description=DRACO — Book-Forged Code Oracle +After=network.target + +[Service] +Type=simple +WorkingDirectory=/opt/draco +ExecStart=/opt/draco/venv/bin/gunicorn -w 2 --threads 8 -b 127.0.0.1:8012 --timeout 300 app:app +Restart=always +RestartSec=5 +Environment=PYTHONUNBUFFERED=1 + +[Install] +WantedBy=multi-user.target diff --git a/deploy/nginx-draco.conf b/deploy/nginx-draco.conf new file mode 100644 index 0000000..1b63c9c --- /dev/null +++ b/deploy/nginx-draco.conf @@ -0,0 +1,17 @@ +server { + listen 80; + server_name draco.thetempleofdoom.com _; + + access_log /var/log/nginx/draco.access.log; + error_log /var/log/nginx/draco.error.log; + + location / { + proxy_pass http://127.0.0.1:8012; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header CF-Connecting-IP $http_cf_connecting_ip; + proxy_buffering off; # SSE + proxy_cache off; + proxy_read_timeout 300s; + } +} diff --git a/draco_core.py b/draco_core.py new file mode 100644 index 0000000..295b245 --- /dev/null +++ b/draco_core.py @@ -0,0 +1,400 @@ +#!/usr/bin/env python3 +"""draco core — RAG engine + monetization for DRACO (coding & hacking book oracle).""" +import json, os, pickle, re, sqlite3, threading, time, secrets + +APP_DIR = os.path.dirname(os.path.abspath(__file__)) +BOOKS_DIR = "/opt/books" +DB_PATH = os.path.join(APP_DIR, "draco.db") + +def load_config(): + with open(os.path.join(APP_DIR, "config.json")) as f: + return json.load(f) + +CFG = load_config() + +# ---------------------------------------------------------------- database +SCHEMA = """ +CREATE TABLE IF NOT EXISTS users ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + email VARCHAR(255) UNIQUE NOT NULL, + api_key VARCHAR(64) UNIQUE NOT NULL, + credits INTEGER DEFAULT 0, + free_used INTEGER DEFAULT 0, + is_admin INTEGER DEFAULT 0, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + last_used_at TIMESTAMP +); +CREATE TABLE IF NOT EXISTS usage_log ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER, + api_key VARCHAR(64), + ip VARCHAR(64), + kind VARCHAR(32), + detail VARCHAR(255), + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE TABLE IF NOT EXISTS payments ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER, + invoice_id VARCHAR(128) UNIQUE, + amount_usd REAL, + credits INTEGER, + status VARCHAR(32) DEFAULT 'pending', + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + settled_at TIMESTAMP +); +CREATE TABLE IF NOT EXISTS anon_usage ( + ip VARCHAR(64) PRIMARY KEY, + day VARCHAR(16), + count INTEGER DEFAULT 0 +); +""" + +def db(): + c = sqlite3.connect(DB_PATH, timeout=15) + c.row_factory = sqlite3.Row + return c + +def init_db(): + c = db() + c.executescript(SCHEMA) + c.execute("INSERT OR IGNORE INTO users (email, api_key, credits, is_admin) VALUES (?,?,?,1)", + ("admin@draco.local", CFG["admin_key"], 999999)) + c.commit() + c.close() + +init_db() # module level: runs under gunicorn too + +# ---------------------------------------------------------------- library index +_lock = threading.Lock() +_idx = None + +def get_index(): + global _idx + with _lock: + if _idx is None: + with open(os.path.join(BOOKS_DIR, "index", "index.pkl"), "rb") as f: + _idx = pickle.load(f) + with open(os.path.join(BOOKS_DIR, "index", "chunks.json")) as f: + _idx["chunks"] = json.load(f) + with open(os.path.join(BOOKS_DIR, "manifest.json")) as f: + manifest = json.load(f) + _idx["books"] = {m["id"]: m for m in manifest} + return _idx + +def get_stats(): + try: + with open(os.path.join(BOOKS_DIR, "book_count.json")) as f: + return json.load(f) + except Exception: + return {"books": 0, "chunks": 0, "chars": 0} + +_STEM = None +def _stemmer(): + global _STEM + if _STEM is None: + from nltk.stem import SnowballStemmer + _STEM = SnowballStemmer("english") + return _STEM + +_STOP = set("""a an and are as at be by for from has have he her his how i in is it its of on or she that the this to was were what when where which who will with you your our we they them them not no but can could should would may might must do does did done if then than into over under about after before between during through against without within""".split()) + +def toks(s): + st = _stemmer() + return [st.stem(w) for w in re.findall(r"[a-z0-9_#+.-]{2,}", s.lower()) if w not in _STOP] + +def search_library(query, k=6): + idx = get_index() + scores = idx["bm25"].get_scores(toks(query)) + order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k] + out = [] + for i in order: + if scores[i] <= 0: + continue + rec = idx["chunks"][i] + book = idx["books"].get(rec["book"], {}) + out.append({"score": round(float(scores[i]), 2), "book_id": rec["book"], + "title": book.get("title", "?"), "category": book.get("category", "?"), + "chunk": rec["chunk"], "text": idx["texts"][i]}) + return out + +# ---------------------------------------------------------------- RAG prompt +SYSTEM = ( + "You are DRACO, a master coding and security tutor. You answer ONLY from the " + "book excerpts provided. Cite sources inline as [n]. If the excerpts do not " + "contain the answer, say so and answer from general expertise, marked " + "(general knowledge). Be precise, technical, and give working code. " + "Answer directly with no reasoning preamble and no meta commentary." +) + +def build_prompt(question, k=6): + hits = search_library(question, k=k) + if hits: + blocks = [] + for n, h in enumerate(hits, 1): + blocks.append(f"[{n}] {h['title']} ({h['category']})\n{h['text'][:1400]}") + ctx = "\n\n".join(blocks) + prompt = f"{SYSTEM}\n\nBOOK EXCERPTS:\n{ctx}\n\nQUESTION: {question}\n\nANSWER (cite [n]):" + else: + prompt = f"{SYSTEM}\n\n(No book excerpts matched — answer from general expertise, marked.)\n\nQUESTION: {question}\n\nANSWER:" + return prompt, hits + +# ---------------------------------------------------------------- ollama +import requests as http + +def strip_cot(text): + text = re.sub(r"(?s).*?", "", text) + if "" in text: + text = text.split("")[-1] + if "" in text: # unbalanced opener: keep what follows + text = text.split("")[-1] + return text.strip() + +def ask_ollama(prompt): + r = http.post(f"{CFG['ollama_url']}/api/generate", + json={"model": CFG["model"], "prompt": prompt, "stream": False, + "options": {"temperature": 0.4, "num_predict": 700, + "num_ctx": CFG.get("num_ctx", 8192)}}, + timeout=180) + r.raise_for_status() + return strip_cot(r.json().get("response", "")) + +class CoTFilter: + """Streaming filter that drops ... blocks token-by-token.""" + OPEN, CLOSE = "", "" + + def __init__(self): + self.buf, self.in_think = "", False + + def feed(self, s): + self.buf += s + out = "" + while True: + if self.in_think: + i = self.buf.find(self.CLOSE) + if i == -1: + keep = min(len(self.buf), len(self.CLOSE) - 1) + self.buf = self.buf[-keep:] if keep else "" + return out + self.in_think = False + self.buf = self.buf[i + len(self.CLOSE):] + else: + i = self.buf.find(self.OPEN) + if i == -1: + keep = min(len(self.buf), len(self.OPEN) - 1) + cut = len(self.buf) - keep + out += self.buf[:cut] + self.buf = self.buf[cut:] + return out + out += self.buf[:i] + self.in_think = True + self.buf = self.buf[i + len(self.OPEN):] + + def flush(self): + out, self.buf = ("" if self.in_think else self.buf), "" + return out + +def stream_ollama(prompt): + r = http.post(f"{CFG['ollama_url']}/api/generate", + json={"model": CFG["model"], "prompt": prompt, "stream": True, + "options": {"temperature": 0.4, "num_predict": 700, + "num_ctx": CFG.get("num_ctx", 8192)}}, + timeout=(5, None), stream=True) + flt = CoTFilter() + for line in r.iter_lines(): + if line: + tok = json.loads(line).get("response", "") + if tok: + piece = flt.feed(tok) + if piece: + yield piece + tail = flt.flush() + if tail: + yield tail + +# ---------------------------------------------------------------- auth + metering +def auth_user(req): + key = req.headers.get("X-API-Key") or (req.json or {}).get("api_key") if req.is_json else req.headers.get("X-API-Key") + if not key: + return None, "API key required (X-API-Key header or api_key field)." + c = db() + u = c.execute("SELECT * FROM users WHERE api_key=?", (key,)).fetchone() + c.close() + if not u: + return None, "Invalid API key." + return dict(u), None + +def consume(user, kind, detail="", ip=""): + c = db() + if user and not user.get("is_admin"): + cur = c.execute("SELECT credits, free_used FROM users WHERE id=?", (user["id"],)).fetchone() + row = c.execute("SELECT * FROM users WHERE id=?", (user["id"],)).fetchone() + month = time.strftime("%Y-%m") + if row["free_used"] < CFG["free_monthly"]: + c.execute("UPDATE users SET free_used=free_used+1, last_used_at=CURRENT_TIMESTAMP WHERE id=?", (user["id"],)) + elif row["credits"] > 0: + c.execute("UPDATE users SET credits=credits-1, last_used_at=CURRENT_TIMESTAMP WHERE id=?", (user["id"],)) + else: + c.close() + return False, "Out of credits. Free tier: %d/mo. Buy more at %s/pricing" % ( + CFG["free_monthly"], CFG["base_url"]) + c.execute("INSERT INTO usage_log (user_id, api_key, ip, kind, detail) VALUES (?,?,?,?,?)", + (user["id"], user["api_key"], ip, kind, detail[:200])) + else: + c.execute("INSERT INTO usage_log (api_key, ip, kind, detail) VALUES (?,?,?,?)", + (user["api_key"] if user else None, ip, kind, detail[:200])) + c.commit() + c.close() + return True, "ok" + +def anon_allowed(ip): + """10 chat messages per IP per day from the web UI.""" + day = time.strftime("%Y-%m-%d") + c = db() + row = c.execute("SELECT * FROM anon_usage WHERE ip=?", (ip,)).fetchone() + if not row or row["day"] != day: + c.execute("INSERT OR REPLACE INTO anon_usage (ip, day, count) VALUES (?,?,0)", (ip, day)) + c.commit() + c.close() + return True + ok = row["count"] < CFG["anon_daily"] + if ok: + c.execute("UPDATE anon_usage SET count=count+1 WHERE ip=?", (ip,)) + c.commit() + c.close() + return ok + +def anon_remaining(ip): + day = time.strftime("%Y-%m-%d") + c = db() + row = c.execute("SELECT * FROM anon_usage WHERE ip=? AND day=?", (ip, day)).fetchone() + c.close() + used = row["count"] if row else 0 + return max(0, CFG["anon_daily"] - used) + +def client_ip(req): + return (req.headers.get("CF-Connecting-IP") or req.headers.get("X-Forwarded-For", req.remote_addr or "?")).split(",")[0].strip() + +def signup(email): + email = email.strip().lower() + if not re.match(r"^[^@\s]+@[^@\s]+\.[^@\s]+$", email): + return None, "Valid email required." + key = "sk-draco-" + secrets.token_hex(20) + c = db() + try: + c.execute("INSERT INTO users (email, api_key) VALUES (?,?)", (email, key)) + c.commit() + except sqlite3.IntegrityError: + c.rollback() + row = c.execute("SELECT api_key FROM users WHERE email=?", (email,)).fetchone() + c.close() + return row["api_key"], "existing" + c.close() + return key, "new" + +# ---------------------------------------------------------------- llms.txt content +def load_llms_txt(): + s = get_stats() + plans = ", ".join(f"{name}: ${p['usd']} = {p['credits']} credits" for name, p in CFG["plans"].items()) + return f"""# DRACO — The Book-Forged Code Oracle + +> DRACO answers coding, Linux, systems, and security questions with answers grounded +> (RAG, BM25) in a private library of {s['books']} real technical books — programming +> languages, kernels, networks, exploitation, red-team tradecraft — {s['chunks']:,} +> indexed passages, {s['chars']/1e6:.0f}M characters. Every answer cites the books it +> came from as [n] with a sources array. Talk to it in the browser, call the REST API, +> or summon it as an MCP tool from any agent. + +## Core endpoints +- POST {CFG['base_url']}/api/ask — body {{"q": "...", "api_key": "..."}} → {{"answer", "sources[]"}} (RAG + citations) +- GET {CFG['base_url']}/api/search?q=...&k=8 — BM25 passage search across all books (header X-API-Key) +- POST {CFG['base_url']}/api/signup — body {{"email": "..."}} → free API key ({CFG['free_monthly']} credits/month) +- GET {CFG['base_url']}/api/my-usage — credits and recent calls (header X-API-Key) +- POST {CFG['base_url']}/api/create-invoice — body {{"plan": "..."}} → BTCPay checkout_url (Bitcoin only) +- GET {CFG['base_url']}/health — liveness + model + library stats + +## MCP (streamable-http) +- {CFG['base_url']}/mcp — tools: draco_ask(question), draco_search(query,k), draco_status() + +## Web UI +- {CFG['base_url']}/ — homepage + live chat (no key needed, {CFG['anon_daily']}/day per IP) +- {CFG['base_url']}/library — the book collection +- {CFG['base_url']}/pricing — {plans} +- {CFG['base_url']}/api — signup + curl examples + +## Auth +- X-API-Key header (or api_key JSON field). Free tier: signup with email. +- Admin/wholesale: contact drjones@thetempleofdoom.com. + +## Model +- {CFG['model']} on bare-metal Ollama · retrieval: BM25, {CFG.get('rag_k',6)} passages/answer +""" + + +def load_llms_full(): + return load_llms_txt() + """ +## Example: ask with curl + curl -s -X POST {b}/api/ask -H 'Content-Type: application/json' \\ + -d '{{"q": "How does a buffer overflow exploit get written?", "api_key": "sk-draco-..."}}' + +## Example: search passages + curl -s "{b}/api/search?q=port+scanning+with+nmap&k=5" -H 'X-API-Key: sk-draco-...' + +## Example: MCP initialize + POST {b}/mcp Accept: application/json, text/event-stream + {{"jsonrpc":"2.0","id":1,"method":"initialize","params":{{"protocolVersion":"2025-03-26", + "capabilities":{{}},"clientInfo":{{"name":"agent","version":"1"}}}}}} + +## Policy +- Answers cite books [n]; unmatched questions are marked (general knowledge). +- Security content is for education and authorized testing. +""".format(b=CFG["base_url"]) + +# ---------------------------------------------------------------- BTCPay +def create_invoice(user, plan): + plans = CFG["plans"] + if plan not in plans: + return None, "Unknown plan." + p = plans[plan] + b = CFG["btcpay"] + if not b.get("store_id"): + return None, "Payments temporarily offline." + try: + r = http.post(f"{b['url']}/api/v1/stores/{b['store_id']}/invoices", + headers={"Authorization": f"token {b['api_key']}"}, + json={"amount": str(p["usd"]), "currency": "USD", + "metadata": {"userId": user["id"], "credits": p["credits"], + "orderId": f"draco-{user['id']}-{int(time.time())}"}, + "checkout": {"redirectURL": f"{CFG['base_url']}/pricing"}}, + timeout=20, verify=False) + r.raise_for_status() + inv = r.json() + except Exception as e: + return None, f"Invoice error: {e}" + c = db() + c.execute("INSERT INTO payments (user_id, invoice_id, amount_usd, credits) VALUES (?,?,?,?)", + (user["id"], inv["id"], p["usd"], p["credits"])) + c.commit() + c.close() + return inv, None + +def handle_webhook(body, header_sig_ok): + if not header_sig_ok: + return False + etype = body.get("type", "") + invoice_id = body.get("invoiceId", "") + if etype not in ("InvoiceSettled", "InvoiceProcessing"): + return True + c = db() + p = c.execute("SELECT * FROM payments WHERE invoice_id=?", (invoice_id,)).fetchone() + if not p: + c.close() + return True + if etype == "InvoiceSettled": + c.execute("UPDATE payments SET status='settled', settled_at=CURRENT_TIMESTAMP WHERE invoice_id=?", (invoice_id,)) + c.execute("UPDATE users SET credits=credits+? WHERE id=?", (p["credits"], p["user_id"])) + else: + c.execute("UPDATE payments SET status='processing' WHERE invoice_id=?", (invoice_id,)) + c.commit() + c.close() + return True diff --git a/index.log b/index.log new file mode 100644 index 0000000..4063cf7 --- /dev/null +++ b/index.log @@ -0,0 +1,5 @@ +index: 1855 usable books from manifest +index: 116558 chunks from 1855 books +index: tokenizing... +index: fitting BM25... +index DONE: {'books': 1855, 'chunks': 116558, 'chars': 354461331, 'built': '2026-10-02T00:30:12'} in 204s diff --git a/ingest.log b/ingest.log new file mode 100644 index 0000000..8e7bd7d --- /dev/null +++ b/ingest.log @@ -0,0 +1,79 @@ +ingest: 1884 books queued + 25/1884 converted (23 with text) + 50/1884 converted (48 with text) + 75/1884 converted (70 with text) + 100/1884 converted (94 with text) + 125/1884 converted (119 with text) + 150/1884 converted (144 with text) + 175/1884 converted (169 with text) + 200/1884 converted (192 with text) + 225/1884 converted (217 with text) + 250/1884 converted (242 with text) + 275/1884 converted (267 with text) + 300/1884 converted (291 with text) + 325/1884 converted (316 with text) + 350/1884 converted (341 with text) + 375/1884 converted (366 with text) + 400/1884 converted (391 with text) + 425/1884 converted (414 with text) + 450/1884 converted (439 with text) + 475/1884 converted (463 with text) + 500/1884 converted (488 with text) + 525/1884 converted (512 with text) + 550/1884 converted (536 with text) + 575/1884 converted (561 with text) + 600/1884 converted (586 with text) + 625/1884 converted (610 with text) + 650/1884 converted (635 with text) + 675/1884 converted (658 with text) + 700/1884 converted (683 with text) + 725/1884 converted (707 with text) + 750/1884 converted (731 with text) + 775/1884 converted (756 with text) + 800/1884 converted (781 with text) + 825/1884 converted (806 with text) + 850/1884 converted (831 with text) + 875/1884 converted (856 with text) + 900/1884 converted (881 with text) + 925/1884 converted (906 with text) + 950/1884 converted (931 with text) + 975/1884 converted (956 with text) + 1000/1884 converted (981 with text) + 1025/1884 converted (1005 with text) + 1050/1884 converted (1030 with text) + 1075/1884 converted (1055 with text) + 1100/1884 converted (1080 with text) + 1125/1884 converted (1104 with text) + 1150/1884 converted (1128 with text) + 1175/1884 converted (1152 with text) + 1200/1884 converted (1177 with text) + 1225/1884 converted (1202 with text) + 1250/1884 converted (1227 with text) + 1275/1884 converted (1252 with text) + 1300/1884 converted (1277 with text) + 1325/1884 converted (1302 with text) + 1350/1884 converted (1327 with text) + 1375/1884 converted (1352 with text) + 1400/1884 converted (1377 with text) + 1425/1884 converted (1402 with text) + 1450/1884 converted (1427 with text) + 1475/1884 converted (1452 with text) + 1500/1884 converted (1477 with text) + 1525/1884 converted (1502 with text) + 1550/1884 converted (1527 with text) + 1575/1884 converted (1552 with text) + 1600/1884 converted (1577 with text) + 1625/1884 converted (1602 with text) + 1650/1884 converted (1627 with text) + 1675/1884 converted (1652 with text) + 1700/1884 converted (1677 with text) + 1725/1884 converted (1702 with text) + 1750/1884 converted (1723 with text) + 1775/1884 converted (1747 with text) + 1800/1884 converted (1771 with text) + 1825/1884 converted (1796 with text) + 1850/1884 converted (1821 with text) + 1875/1884 converted (1846 with text) + 1884/1884 converted (1855 with text) +ingest DONE: 1855/1884 books usable, 354.5M chars total +PIPELINE_DONE diff --git a/ingest.py b/ingest.py new file mode 100644 index 0000000..aa2e5f7 --- /dev/null +++ b/ingest.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""draco ingest — convert every book (pdf/chm/djvu/djv/txt) to plain text + manifest. +Run on the CT: python3 ingest.py [--books /opt/books] [--workers 4] +Output: /opt/books/text//.txt + /opt/books/manifest.json +""" +import argparse, concurrent.futures as cf, hashlib, html as htmllib, json, os, re, shutil, subprocess, sys, tempfile, traceback + +BOOKS_DIR = "/opt/books" +TEXT_DIR = os.path.join(BOOKS_DIR, "text") + +def slugify(name: str) -> str: + s = re.sub(r"\.(pdf|chm|djvu|djv|txt|epub)$", "", name, flags=re.I) + s = s.replace("&", " and ").replace("+", " plus ") + s = re.sub(r"[^A-Za-z0-9._-]+", "_", s).strip("._") + return s[:110] or "book" + +def clean_text(t: str) -> str: + t = t.replace("\x00", "") + t = re.sub(r"[ \t]+", " ", t) + t = re.sub(r"\n{3,}", "\n\n", t) + return t.strip() + +def html_to_text(h: str) -> str: + h = re.sub(r"(?is)<(script|style)[^>]*>.*?", " ", h) + h = re.sub(r"(?i)|

||||", "\n", h) + h = re.sub(r"<[^>]+>", " ", h) + return htmllib.unescape(h) + +def convert(path: str, fmt: str, out_path: str) -> tuple[int, str]: + """Returns (n_chars, err).""" + if fmt == "txt": + shutil.copyfile(path, out_path) + return os.path.getsize(out_path), "" + if fmt == "pdf": + r = subprocess.run(["pdftotext", "-q", "-layout", path, out_path], capture_output=True, timeout=300) + return (os.path.getsize(out_path) if os.path.exists(out_path) else 0), (r.stderr.decode()[:200] if r.returncode else "") + if fmt == "djvu": + r = subprocess.run(["djvutxt", path], capture_output=True, timeout=300) + txt = clean_text(r.stdout.decode("utf-8", "replace")) + open(out_path, "w").write(txt) + return len(txt), "" + if fmt == "chm": + tmp = tempfile.mkdtemp(prefix="chm_") + try: + r = subprocess.run(["extract_chmLib", path, tmp], capture_output=True, timeout=300) + parts = [] + for root, _, files in os.walk(tmp): + for f in sorted(files): + if f.lower().endswith((".html", ".htm", ".txt")): + try: + raw = open(os.path.join(root, f), "rb").read() + for enc in ("utf-8", "cp1252", "latin-1"): + try: + parts.append(html_to_text(raw.decode(enc))); break + except UnicodeDecodeError: + continue + except Exception: + pass + txt = clean_text("\n\n".join(parts)) + open(out_path, "w").write(txt) + return len(txt), "" + finally: + shutil.rmtree(tmp, ignore_errors=True) + return 0, "unsupported" + +def process(task): + path, category, books_dir, text_dir = task + fmt = path.rsplit(".", 1)[-1].lower() + base = slugify(os.path.basename(path)) + bid = hashlib.sha1(f"{category}/{base}".encode()).hexdigest()[:10] + title = re.sub(r"\.(pdf|chm|djvu|djv|txt|epub)$", "", os.path.basename(path), flags=re.I) + title = re.sub(r"[_\.]+", " ", title).strip() + out_rel = f"{category}/{bid}_{base}.txt" + out_abs = os.path.join(text_dir, out_rel) + os.makedirs(os.path.dirname(out_abs), exist_ok=True) + n, err = 0, "" + try: + n, err = convert(path, fmt, out_abs) + except subprocess.TimeoutExpired: + err = "timeout" + except Exception: + err = traceback.format_exc(limit=1).splitlines()[-1] + if n < 200 and os.path.exists(out_abs): + os.remove(out_abs) # scanned/no-text junk + n = 0 + return {"id": bid, "title": title, "category": category, "format": fmt, + "source": os.path.relpath(path, books_dir), "txt": out_rel if n else None, + "chars": n, "bytes": os.path.getsize(path), "err": err[:200]} + +def main(): + global BOOKS_DIR, TEXT_DIR + ap = argparse.ArgumentParser() + ap.add_argument("--books", default=BOOKS_DIR) + ap.add_argument("--workers", type=int, default=4) + a = ap.parse_args() + BOOKS_DIR, TEXT_DIR = a.books, os.path.join(a.books, "text") + os.makedirs(TEXT_DIR, exist_ok=True) + tasks = [] + for root, dirs, files in os.walk(BOOKS_DIR): + dirs[:] = [d for d in dirs if not d.startswith(".") and d != "text"] + for f in files: + if f.startswith("._") or f == ".gitignore": + continue + if f.rsplit(".", 1)[-1].lower() in ("pdf", "chm", "djvu", "djv", "txt"): + rel = os.path.relpath(os.path.join(root, f), BOOKS_DIR) + category = rel.split(os.sep)[0] + tasks.append((os.path.join(root, f), category, BOOKS_DIR, TEXT_DIR)) + print(f"ingest: {len(tasks)} books queued", flush=True) + manifest, done = [], 0 + with cf.ProcessPoolExecutor(max_workers=a.workers) as ex: + for rec in ex.map(process, tasks): + manifest.append(rec) + done += 1 + if done % 25 == 0 or done == len(tasks): + ok = sum(1 for m in manifest if m["chars"]) + print(f" {done}/{len(tasks)} converted ({ok} with text)", flush=True) + manifest.sort(key=lambda m: (m["category"], m["title"].lower())) + with open(os.path.join(BOOKS_DIR, "manifest.json"), "w") as f: + json.dump(manifest, f, indent=1) + ok = [m for m in manifest if m["chars"]] + print(f"ingest DONE: {len(ok)}/{len(manifest)} books usable, " + f"{sum(m['chars'] for m in ok)/1e6:.1f}M chars total", flush=True) + +if __name__ == "__main__": + main() diff --git a/mcp_client.py b/mcp_client.py new file mode 100644 index 0000000..86bbf48 --- /dev/null +++ b/mcp_client.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""DRACO stdio MCP client — bridges DRACO's remote /mcp into any stdio MCP host. + +Register with Claude Desktop / Hermes: + draco-search: + command: python3 + args: [/path/to/mcp_client.py] + env: + DRACO_MCP_URL: "https://draco.thetempleofdoom.com/mcp" +""" +import asyncio, json, os, sys +import urllib.request + +MCP_URL = os.environ.get("DRACO_MCP_URL", "https://draco.thetempleofdoom.com/mcp") + +def rpc(method, params=None, session=None, notify=False): + body = {"jsonrpc": "2.0", "method": method} + if params is not None: + body["params"] = params + if not notify: + body["id"] = 1 + req = urllib.request.Request(MCP_URL, data=json.dumps(body).encode(), + headers={"Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + **({"mcp-session-id": session} if session else {})}) + resp = urllib.request.urlopen(req, timeout=120) + sid = resp.headers.get("mcp-session-id") + raw = resp.read().decode() + if not notify: + for line in raw.splitlines(): + if line.startswith("data: "): + return json.loads(line[6:]), sid + return (json.loads(raw) if raw.strip() else {}), sid + return None, sid + +def parse_sse_text(raw): + for line in raw.splitlines(): + if line.startswith("data: "): + return json.loads(line[6:]) + return json.loads(raw) if raw.strip() else {} + +def call_tool(name, args): + _, sid = rpc("initialize", {"protocolVersion": "2025-03-26", + "capabilities": {}, + "clientInfo": {"name": "draco-stdio", "version": "1"}}) + rpc("notifications/initialized", {}, session=sid, notify=True) + body = json.dumps({"jsonrpc": "2.0", "id": 2, "method": "tools/call", + "params": {"name": name, "arguments": args}}).encode() + req = urllib.request.Request(MCP_URL, data=body, + headers={"Content-Type": "application/json", + "Accept": "application/json, text/event-stream", + "mcp-session-id": sid}) + resp = urllib.request.urlopen(req, timeout=180) + result = parse_sse_text(resp.read().decode()) + out = result.get("result", {}).get("content", []) + return "\n".join(c.get("text", "") for c in out if c.get("type") == "text") + +async def main(): + from mcp.server import Server + from mcp.server.stdio import stdio_server + + server = Server("draco") + + @server.tool() + async def draco_ask(question: str) -> str: + """Ask DRACO a coding/security question — grounded in hundreds of real books, cited.""" + return call_tool("draco_ask", {"question": question}) + + @server.tool() + async def draco_search(query: str, k: int = 8) -> str: + """Search DRACO's book library (BM25) for passages.""" + return call_tool("draco_search", {"query": query, "k": k}) + + @server.tool() + async def draco_status() -> str: + """DRACO health + library size.""" + return call_tool("draco_status", {}) + + async with stdio_server() as (read, write): + await server.run(read, write, server.create_initialization_options()) + +if __name__ == "__main__": + if len(sys.argv) > 2 and sys.argv[1] == "--direct": + print(call_tool(sys.argv[2], json.loads(sys.argv[3]) if len(sys.argv) > 3 else {})) + else: + asyncio.run(main()) diff --git a/pages.py b/pages.py new file mode 100644 index 0000000..ea5f5b6 --- /dev/null +++ b/pages.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""DRACO theme part 1: base CSS, nav, homepage (hero + chat).""" + +BASE_CSS = """ +:root{--bg:#07090e;--panel:#0c1017;--panel2:#111725;--line:#1c2436;--txt:#dbe4f0; +--dim:#7d8aa0;--gold:#e8b64c;--gold2:#f5d78a;--green:#3ddc84;--red:#ff5c5c;--mono:ui-monospace,'JetBrains Mono','Fira Code',Menlo,monospace} +*{margin:0;padding:0;box-sizing:border-box} +body{background:var(--bg);color:var(--txt);font-family:var(--mono);font-size:15px;line-height:1.6; +background-image:radial-gradient(ellipse 80% 50% at 50% -10%,rgba(232,180,76,.07),transparent)} +a{color:var(--gold);text-decoration:none}a:hover{color:var(--gold2)} +nav{display:flex;gap:26px;align-items:center;padding:16px 5vw;border-bottom:1px solid var(--line); +position:sticky;top:0;background:rgba(7,9,14,.92);backdrop-filter:blur(8px);z-index:50} +nav .logo{font-weight:700;font-size:17px;color:var(--gold);letter-spacing:2px} +nav .logo small{color:var(--dim);font-weight:400;letter-spacing:0;margin-left:8px} +nav a{color:var(--dim);font-size:13px;text-transform:uppercase;letter-spacing:1px} +nav a.on,nav a:hover{color:var(--gold)} +main{max-width:1080px;margin:0 auto;padding:40px 5vw 80px} +h1{font-size:34px;line-height:1.2;margin-bottom:10px} +h2{font-size:20px;margin:34px 0 12px;color:var(--gold)} +.dim{color:var(--dim)}.gold{color:var(--gold)}.green{color:var(--green)} +.hero{padding:36px 0 10px} +.badge{display:inline-block;border:1px solid var(--gold);color:var(--gold);border-radius:999px; +padding:3px 14px;font-size:12px;letter-spacing:2px;text-transform:uppercase;margin-bottom:18px} +.stats{display:flex;flex-wrap:wrap;gap:14px;margin:22px 0 8px} +.stat{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:14px 20px;min-width:150px} +.stat b{display:block;font-size:22px;color:var(--gold)} +.stat span{font-size:11px;color:var(--dim);text-transform:uppercase;letter-spacing:1px} +.panel{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:22px;margin-top:18px} +.chat{display:flex;flex-direction:column;height:460px;margin-top:20px} +.chatlog{flex:1;overflow-y:auto;padding:16px;background:var(--bg);border:1px solid var(--line);border-radius:10px} +.msg{margin-bottom:16px;white-space:pre-wrap;word-wrap:break-word} +.msg.user{color:var(--gold2)} +.msg.assistant{color:var(--txt)} +.msg .who{font-size:11px;color:var(--dim);text-transform:uppercase;letter-spacing:1px;display:block;margin-bottom:3px} +.srcs{font-size:12px;color:var(--dim);border-left:2px solid var(--gold);padding-left:10px;margin-top:8px} +.cursor{display:inline-block;width:8px;height:15px;background:var(--gold);animation:blink 1s steps(1) infinite;vertical-align:text-bottom} +@keyframes blink{50%{opacity:0}} +.chatrow{display:flex;gap:10px;margin-top:12px} +.chatrow input{flex:1;background:var(--panel2);border:1px solid var(--line);border-radius:8px; +padding:12px 14px;color:var(--txt);font-family:var(--mono);font-size:14px;outline:none} +.chatrow input:focus{border-color:var(--gold)} +button,.btn{background:var(--gold);color:#14100a;border:none;border-radius:8px;padding:12px 22px; +font-family:var(--mono);font-weight:700;font-size:14px;cursor:pointer;letter-spacing:1px} +button:hover,.btn:hover{background:var(--gold2)} +button:disabled{opacity:.5;cursor:not-allowed} +.btn.ghost{background:transparent;border:1px solid var(--gold);color:var(--gold)} +.chips{display:flex;flex-wrap:wrap;gap:8px;margin-top:10px} +.chip{background:var(--panel2);border:1px solid var(--line);color:var(--dim);border-radius:999px; +padding:6px 14px;font-size:12px;cursor:pointer} +.chip:hover{border-color:var(--gold);color:var(--gold)} +table{width:100%;border-collapse:collapse;margin:14px 0} +th,td{text-align:left;padding:10px 12px;border-bottom:1px solid var(--line);font-size:13px} +th{color:var(--dim);text-transform:uppercase;font-size:11px;letter-spacing:1px} +code,pre{background:var(--panel2);border-radius:6px;font-size:13px} +code{padding:2px 6px} +pre{padding:14px;overflow-x:auto;border:1px solid var(--line);margin:10px 0} +.grid{display:grid;grid-template-columns:repeat(auto-fill,minmax(240px,1fr));gap:12px} +.card{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:14px} +.card b{color:var(--gold);font-size:13px} +.card span{display:block;color:var(--dim);font-size:11px;margin-top:4px;text-transform:uppercase;letter-spacing:1px} +footer{border-top:1px solid var(--line);margin-top:60px;padding:26px 5vw;display:flex; +justify-content:space-between;color:var(--dim);font-size:12px;flex-wrap:wrap;gap:10px} +.kbd{border:1px solid var(--line);border-bottom-width:2px;border-radius:5px;padding:1px 6px;font-size:12px;background:var(--panel2)} +""" + +NAV = """ +Oracle +Library +API +Pricing +llms.txt +☕ BMAC""" + +PAGE_HOME = """ +
+ ◈ Grounded in real books — not vibes +

The smartest coding AI you can prove.

+

Most AI assistants guess. DRACO retrieves from __STATS__ — + languages, kernels, networks, exploitation, red-team tradecraft — and shows you the exact + books behind every answer. Check the citations. That's the proof.

+
+
{{ books }}real books indexed
+
{{ chunks }}retrievable passages
+
{{ mb }}Mchars of expert text
+
<2sretrieval + answer start
+
+
+ +
+

◈ Interrogate the Oracle

+

{{ free_day }} free questions a day from the browser, no signup. + Coding · Linux · networking · security tradecraft.

+
+
+
dracoAsk me anything from the shelves. I cite my sources — hover the gold bar under each answer to see which books I pulled from.
+
+
+ + +
+
+ explain stack buffer overflows with a poc + how do I harden ssh on a public server? + write a python port scanner with threads + what is a kernel panic and how do I debug one? + nmap SYN scan vs connect scan — when and why +
+

get a free API key →

+
+
+ +

◈ Why "book-forged" is different

+
+
Every claim has a receiptanswers cite [n] — click through to the source book in /library
+
No 2021 knowledge cutofffundamentals from the canon: K&R, Tanenbaum, the shellcoders' handbook
+
Agentic by designMCP server + REST API — summon DRACO from Claude, GPT, or any agent
+
Self-hosted, no telemetryruns on bare metal; questions never leave the house
+
+ +

◈ Summon it as a tool

+
{
+  "mcpServers": {
+    "draco": { "url": "__BASE__/mcp" }
+  }
+}
+

Drop that in any MCP client config. Tools: draco_ask, draco_search, draco_status.

+ + + + +""" diff --git a/pages2.py b/pages2.py new file mode 100644 index 0000000..db6f583 --- /dev/null +++ b/pages2.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""DRACO theme part 2: library, pricing, api pages. Jinja vars injected by app.render().""" + +PAGE_LIBRARY = """ +

The Hoard

+

__STATS__. Every passage DRACO cites comes from these shelves. Downloads are +free for personal use — this is the library, open to the public.

+

◈ Collections

+ + + + +
CollectionContents
library_linuxThe coding canon: C/C++, Python, Perl, PHP, Java, assembly, SQL/databases, web, Unix/Linux internals, networks & security, math, Apache, Windows/M$.
hackerpackThe Hacker Pro Pack: exploitation, malware, network attacks, crypto, wireless, OSINT, lockpicking-of-the-digital-kind, and the classics of the scene.
+

◈ Browse the shelves

+
+ +

+
+ + +""" + +PAGE_PRICING = """ +

Pricing

+

No subscriptions. No accounts-with-email-verification loops. Bitcoin only, +self-hosted BTCPay — no card processor ever sees you or us.

+
+
FREE{{ free_monthly }} API credits / month + {{ free_day }} web questions / day
+
1 credit= 1 ask (RAG + citations) or 1 search
+
+

◈ Credit packs — one-time, never expire

+ + +{% for name, p in plans.items() %} + + +{% endfor %} +
PackPriceCredits$/credit
{{ name }}${{ '%.2f'|format(p.usd) }}{{ p.credits }}${{ '%.3f'|format(p.usd / p.credits) }}
+

+

◈ Why paid tiers exist

+
+
GPU time is realevery answer runs retrieval + local inference on bare metal — electricity isn't free
+
The free tier stays free{{ free_monthly }}/mo forever, no card, email is just your key-recovery address
+
Zero data resalewe sell answers, not you. no telemetry, no logs sold, ever
+
Bitcoin onlypay from any wallet, no chargebacks, no KYC, invoices live ~15 min
+
+ + +""" + +PAGE_API = """ +

API

+

One endpoint, RAG-grounded answers with citations. Free {{ free_monthly }} calls/month. +Works with curl, any HTTP client, or as an MCP tool for your agents.

+ +

◈ Get your key

+
+ + +
+ + +

◈ Ask

+
curl -s -X POST {{ base }}/api/ask \\
+  -H 'Content-Type: application/json' \\
+  -d '{"q": "explain integer overflow with C poc", "api_key": "sk-draco-..."}'
+

→ {"answer": "...", "sources": [{"title": "...", "category": "..."}]}

+ +

◈ Search the library raw

+
curl -s "{{ base }}/api/search?q=tcp+syn+flood&k=5" \\
+  -H 'X-API-Key: sk-draco-...'
+ +

◈ Usage & credits

+
curl -s {{ base }}/api/my-usage -H 'X-API-Key: sk-draco-...'
+ +

◈ MCP (for Claude / GPT / any agent)

+
{
+  "mcpServers": {
+    "draco": { "url": "{{ base }}/mcp" }
+  }
+}
+

Tools: draco_ask · draco_search · draco_status. +Machine-readable docs: llms.txt · openapi.json · +ai-plugin.json

+ +

◈ Rate limits & rules

+ + + + + + +
TierLimits
Web (no key){{ free_day }} questions/day per IP
Free API{{ free_monthly }}/month, then buy credits
Paid credits1 credit = 1 ask/search, never expire
Adminunlimited — contact
+ + +""" diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..7a7806b --- /dev/null +++ b/requirements.txt @@ -0,0 +1,8 @@ +flask +requests +rank_bm25 +nltk +mcp +uvicorn +starlette +gunicorn