Files

drjones 34061a4ac1 AI Research Engine — self-hosted knowledge acquisition system

- 10 MCP tools via thin proxy on MacBook
- Backend REST API + Dashboard on CT 145 Docker
- Services: YaCy crawler, OpenSearch index, Qdrant vectors, Ollama LLM
- All 4 services healthy and verified

2026-08-04 06:02:42 -07:00

4.6 KiB

Raw Blame History

🧠 AI Research Engine

Self-hosted agentic AI search infrastructure — your private research cloud.

AI agents connect via MCP to discover, crawl, index, and synthesize knowledge from the web — all running on your own Proxmox hardware.

Architecture

┌──────────────────────────────────────────────────┐
│  MacBook (thin MCP proxy)                        │
│  server.py → forwards to CT 145 backend          │
└──────────────┬───────────────────────────────────┘
               │ HTTP
┌──────────────▼───────────────────────────────────┐
│  CT 145 (10.30.20.249) — Docker Host             │
│  ┌─────────┐ ┌──────────┐ ┌───────┐ ┌─────────┐ │
│  │  YaCy   │ │OpenSearch│ │ Redis │ │ Backend │ │
│  │  :8090  │ │  :9200   │ │ :6379 │ │  :8000  │ │
│  │ crawl   │ │  index   │ │ cache │ │  API+UI │ │
│  └─────────┘ └──────────┘ └───────┘ └─────────┘ │
└──────────────────────────────────────────────────┘
               │                    │
┌──────────────▼─────┐  ┌──────────▼──────────────┐
│  CT 509 (.68)      │  │  GamingPC (.186)        │
│  Qdrant :6333      │  │  Ollama :11434          │
│  semantic search   │  │  ornith:latest (9B)     │
└────────────────────┘  └─────────────────────────┘

Endpoints

Service	URL	Purpose
Dashboard	http://10.30.20.249:8000	Web UI
Backend API	http://10.30.20.249:8000/api/*	REST API
OpenSearch	http://10.30.20.249:9200	Full-text index
YaCy	http://10.30.20.249:8090	Web crawler
Qdrant	http://10.30.20.68:6333	Vector DB
Ollama	http://10.30.20.186:11434	LLM inference

MCP Tools (10)

Tool	Description
`search_web(query)`	Full-text search across indexed documents
`semantic_search(query)`	Vector search by meaning (Qdrant)
`crawl_url(url)`	Crawl a URL into the index
`crawl_topic(topic)`	Discover + crawl sources for a topic
`research_topic(topic)`	Full pipeline: search → crawl → summarize
`retrieve_document(url)`	Get full indexed document content
`summarize_sources(urls)`	AI summary of multiple sources
`extract_information(url, schema)`	Structured data extraction
`create_report(topic)`	Comprehensive research report
`index_status()`	System health check

Quick Start

# Check status
curl http://10.30.20.249:8000/api/status

# Search
curl "http://10.30.20.249:8000/api/search?q=knowledge+graphs"

# Crawl a URL
curl "http://10.30.20.249:8000/api/crawl?url=https://example.com&depth=1"

# Deep research
curl "http://10.30.20.249:8000/api/research?topic=LED+grow+lights"

# Generate report
curl "http://10.30.20.249:8000/api/report?topic=AI+agents"

Deployment

On CT 145 (10.30.20.249):

# Services
docker run -d --name redis --restart unless-stopped -p 6379:6379 redis:7-alpine
docker run -d --name yacy --restart unless-stopped -p 8090:8090 yacy/yacy_search_server:latest
docker run -d --name opensearch --restart unless-stopped -p 9200:9200 \
  -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" \
  -e "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx2g" opensearchproject/opensearch:2.17.0

# Backend
docker build -t research-backend .
docker run -d --name research-backend --restart unless-stopped -p 8000:8000 \
  --add-host=host.docker.internal:host-gateway \
  -e YACY_URL=http://host.docker.internal:8090 \
  -e OPENSEARCH_URL=http://host.docker.internal:9200 \
  -e QDRANT_URL=http://10.30.20.68:6333 \
  -e OLLAMA_URL=http://10.30.20.186:11434 \
  research-backend

Files

server.py — Thin MCP proxy (runs on MacBook)
backend.py — REST API + Dashboard (runs on CT 145)
docker-compose.yml — Reference compose file
Dockerfile — Backend container build
.env — Service endpoints config

4.6 KiB Raw Blame History