34061a4ac1
- 10 MCP tools via thin proxy on MacBook - Backend REST API + Dashboard on CT 145 Docker - Services: YaCy crawler, OpenSearch index, Qdrant vectors, Ollama LLM - All 4 services healthy and verified
106 lines
4.6 KiB
Markdown
106 lines
4.6 KiB
Markdown
# 🧠 AI Research Engine
|
|
|
|
**Self-hosted agentic AI search infrastructure** — your private research cloud.
|
|
|
|
AI agents connect via MCP to discover, crawl, index, and synthesize knowledge from the web — all running on your own Proxmox hardware.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────┐
|
|
│ MacBook (thin MCP proxy) │
|
|
│ server.py → forwards to CT 145 backend │
|
|
└──────────────┬───────────────────────────────────┘
|
|
│ HTTP
|
|
┌──────────────▼───────────────────────────────────┐
|
|
│ CT 145 (10.30.20.249) — Docker Host │
|
|
│ ┌─────────┐ ┌──────────┐ ┌───────┐ ┌─────────┐ │
|
|
│ │ YaCy │ │OpenSearch│ │ Redis │ │ Backend │ │
|
|
│ │ :8090 │ │ :9200 │ │ :6379 │ │ :8000 │ │
|
|
│ │ crawl │ │ index │ │ cache │ │ API+UI │ │
|
|
│ └─────────┘ └──────────┘ └───────┘ └─────────┘ │
|
|
└──────────────────────────────────────────────────┘
|
|
│ │
|
|
┌──────────────▼─────┐ ┌──────────▼──────────────┐
|
|
│ CT 509 (.68) │ │ GamingPC (.186) │
|
|
│ Qdrant :6333 │ │ Ollama :11434 │
|
|
│ semantic search │ │ ornith:latest (9B) │
|
|
└────────────────────┘ └─────────────────────────┘
|
|
```
|
|
|
|
## Endpoints
|
|
|
|
| Service | URL | Purpose |
|
|
|---------|-----|---------|
|
|
| Dashboard | http://10.30.20.249:8000 | Web UI |
|
|
| Backend API | http://10.30.20.249:8000/api/* | REST API |
|
|
| OpenSearch | http://10.30.20.249:9200 | Full-text index |
|
|
| YaCy | http://10.30.20.249:8090 | Web crawler |
|
|
| Qdrant | http://10.30.20.68:6333 | Vector DB |
|
|
| Ollama | http://10.30.20.186:11434 | LLM inference |
|
|
|
|
## MCP Tools (10)
|
|
|
|
| Tool | Description |
|
|
|------|-------------|
|
|
| `search_web(query)` | Full-text search across indexed documents |
|
|
| `semantic_search(query)` | Vector search by meaning (Qdrant) |
|
|
| `crawl_url(url)` | Crawl a URL into the index |
|
|
| `crawl_topic(topic)` | Discover + crawl sources for a topic |
|
|
| `research_topic(topic)` | Full pipeline: search → crawl → summarize |
|
|
| `retrieve_document(url)` | Get full indexed document content |
|
|
| `summarize_sources(urls)` | AI summary of multiple sources |
|
|
| `extract_information(url, schema)` | Structured data extraction |
|
|
| `create_report(topic)` | Comprehensive research report |
|
|
| `index_status()` | System health check |
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Check status
|
|
curl http://10.30.20.249:8000/api/status
|
|
|
|
# Search
|
|
curl "http://10.30.20.249:8000/api/search?q=knowledge+graphs"
|
|
|
|
# Crawl a URL
|
|
curl "http://10.30.20.249:8000/api/crawl?url=https://example.com&depth=1"
|
|
|
|
# Deep research
|
|
curl "http://10.30.20.249:8000/api/research?topic=LED+grow+lights"
|
|
|
|
# Generate report
|
|
curl "http://10.30.20.249:8000/api/report?topic=AI+agents"
|
|
```
|
|
|
|
## Deployment
|
|
|
|
On CT 145 (10.30.20.249):
|
|
|
|
```bash
|
|
# Services
|
|
docker run -d --name redis --restart unless-stopped -p 6379:6379 redis:7-alpine
|
|
docker run -d --name yacy --restart unless-stopped -p 8090:8090 yacy/yacy_search_server:latest
|
|
docker run -d --name opensearch --restart unless-stopped -p 9200:9200 \
|
|
-e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" \
|
|
-e "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx2g" opensearchproject/opensearch:2.17.0
|
|
|
|
# Backend
|
|
docker build -t research-backend .
|
|
docker run -d --name research-backend --restart unless-stopped -p 8000:8000 \
|
|
--add-host=host.docker.internal:host-gateway \
|
|
-e YACY_URL=http://host.docker.internal:8090 \
|
|
-e OPENSEARCH_URL=http://host.docker.internal:9200 \
|
|
-e QDRANT_URL=http://10.30.20.68:6333 \
|
|
-e OLLAMA_URL=http://10.30.20.186:11434 \
|
|
research-backend
|
|
```
|
|
|
|
## Files
|
|
|
|
- `server.py` — Thin MCP proxy (runs on MacBook)
|
|
- `backend.py` — REST API + Dashboard (runs on CT 145)
|
|
- `docker-compose.yml` — Reference compose file
|
|
- `Dockerfile` — Backend container build
|
|
- `.env` — Service endpoints config
|