Commit Graph

3 Commits

Author SHA1 Message Date
drjones
966b0e8d59 Add NordVPN proxy rotation for geo-distributed web fetching
3 proxies: Tokyo (.154:3128), London (.71:3128), Sydney (.189:3128)
Random rotation per crawl — bypasses geo-blocks and rate limits
2026-08-04 06:20:37 -07:00
drjones
da75b635f4 Add fetch-and-index pipeline: crawl immediately indexes into OpenSearch+Qdrant
- _fetch_and_index() fetches, strips HTML, extracts title/meta, indexes
- crawl_url now indexes immediately (not just submits to YaCy)
- Fixed search: removed invalid refresh param on _search requests
- Auto-categorization by domain (wikipedia→reference, github→software, etc)
- Verified: 5 docs in OpenSearch, 5 vectors in Qdrant
2026-08-04 06:12:05 -07:00
drjones
34061a4ac1 AI Research Engine — self-hosted knowledge acquisition system
- 10 MCP tools via thin proxy on MacBook
- Backend REST API + Dashboard on CT 145 Docker
- Services: YaCy crawler, OpenSearch index, Qdrant vectors, Ollama LLM
- All 4 services healthy and verified
2026-08-04 06:02:42 -07:00