2ac65db518
- _fetch_and_index now extracts img src URLs, filters junk, stores in OpenSearch - get_db() uses PGHOST/PGUSER/etc env vars for Docker compatibility - search endpoint returns images + image_count in hits - Docker run command passes all PG env vars - Discovery script runs via SSH to CT to avoid VPN issues
🧠 AI Research Engine
Self-hosted agentic AI search infrastructure — your private research cloud.
AI agents connect via MCP to discover, crawl, index, and synthesize knowledge from the web — all running on your own Proxmox hardware.
Architecture
┌──────────────────────────────────────────────────┐
│ MacBook (thin MCP proxy) │
│ server.py → forwards to CT 145 backend │
└──────────────┬───────────────────────────────────┘
│ HTTP
┌──────────────▼───────────────────────────────────┐
│ CT 145 (10.30.20.249) — Docker Host │
│ ┌─────────┐ ┌──────────┐ ┌───────┐ ┌─────────┐ │
│ │ YaCy │ │OpenSearch│ │ Redis │ │ Backend │ │
│ │ :8090 │ │ :9200 │ │ :6379 │ │ :8000 │ │
│ │ crawl │ │ index │ │ cache │ │ API+UI │ │
│ └─────────┘ └──────────┘ └───────┘ └─────────┘ │
└──────────────────────────────────────────────────┘
│ │
┌──────────────▼─────┐ ┌──────────▼──────────────┐
│ CT 509 (.68) │ │ GamingPC (.186) │
│ Qdrant :6333 │ │ Ollama :11434 │
│ semantic search │ │ ornith:latest (9B) │
└────────────────────┘ └─────────────────────────┘
Endpoints
| Service | URL | Purpose |
|---|---|---|
| Dashboard | http://10.30.20.249:8000 | Web UI |
| Backend API | http://10.30.20.249:8000/api/* | REST API |
| OpenSearch | http://10.30.20.249:9200 | Full-text index |
| YaCy | http://10.30.20.249:8090 | Web crawler |
| Qdrant | http://10.30.20.68:6333 | Vector DB |
| Ollama | http://10.30.20.186:11434 | LLM inference |
MCP Tools (10)
| Tool | Description |
|---|---|
search_web(query) |
Full-text search across indexed documents |
semantic_search(query) |
Vector search by meaning (Qdrant) |
crawl_url(url) |
Crawl a URL into the index |
crawl_topic(topic) |
Discover + crawl sources for a topic |
research_topic(topic) |
Full pipeline: search → crawl → summarize |
retrieve_document(url) |
Get full indexed document content |
summarize_sources(urls) |
AI summary of multiple sources |
extract_information(url, schema) |
Structured data extraction |
create_report(topic) |
Comprehensive research report |
index_status() |
System health check |
Quick Start
# Check status
curl http://10.30.20.249:8000/api/status
# Search
curl "http://10.30.20.249:8000/api/search?q=knowledge+graphs"
# Crawl a URL
curl "http://10.30.20.249:8000/api/crawl?url=https://example.com&depth=1"
# Deep research
curl "http://10.30.20.249:8000/api/research?topic=LED+grow+lights"
# Generate report
curl "http://10.30.20.249:8000/api/report?topic=AI+agents"
Deployment
On CT 145 (10.30.20.249):
# Services
docker run -d --name redis --restart unless-stopped -p 6379:6379 redis:7-alpine
docker run -d --name yacy --restart unless-stopped -p 8090:8090 yacy/yacy_search_server:latest
docker run -d --name opensearch --restart unless-stopped -p 9200:9200 \
-e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" \
-e "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx2g" opensearchproject/opensearch:2.17.0
# Backend
docker build -t research-backend .
docker run -d --name research-backend --restart unless-stopped -p 8000:8000 \
--add-host=host.docker.internal:host-gateway \
-e YACY_URL=http://host.docker.internal:8090 \
-e OPENSEARCH_URL=http://host.docker.internal:9200 \
-e QDRANT_URL=http://10.30.20.68:6333 \
-e OLLAMA_URL=http://10.30.20.186:11434 \
research-backend
Files
server.py— Thin MCP proxy (runs on MacBook)backend.py— REST API + Dashboard (runs on CT 145)docker-compose.yml— Reference compose fileDockerfile— Backend container build.env— Service endpoints config