add Commando tool catalog as lazy discovery tree (10 categories, ~55 tools) + fix native tool arg parsing + unicode-safe stdout

README.md

@@ -1,15 +1,55 @@
 # METATRON — lightweight Ollama terminal harness
 
 A minimal terminal harness for talking to Ollama (hosted on nightmare `10.30.20.29:11434`)
-with full system access and internet search. Type `metatron` to launch.
+with full system access, internet search, and the Commando tool catalog organized as a
+lazy discovery tree. Type `metatron` to launch.
 
 ## Flow
-1. **Model selection** — fetches the live model list from nightmare, numbered menu.
+1. **Model selection** — fetches the live model list from nightmare, numbered menu
+   (sizes shown, `Enter` reuses the last model).
 2. **Chat** — streaming REPL against the selected model.
-3. **System access** — `!command` runs a shell command directly, and the model can also
-   call `run_command` natively (function calling).
-4. **Search** — `?query` searches the internet (SearXNG), and the model can call
-   `web_search` natively.
+3. **System access** — `!command` runs a shell command directly; the model can also call
+   `run_command` natively (function calling).
+4. **Search** — `?query` searches (SearXNG); `?fetch <url>` reads a page.
+5. **Tool discovery tree** — the Commando toolset is organized by category and discovered
+   lazily, so the model never has all tools dumped into context at once.
+
+## Tool discovery tree
+The catalog (~55 tools across 10 categories) is NOT sent to the model. Instead the model
+gets 4 native tools and drills in on demand:
+
+```
+list_categories            -> see the 10 categories (recon, web, exploit, post, crack,
+                              wireless, sniff, osint, forensics, utils)
+list_tools("<category>")   -> see the tools + usage in one category
+run_command                -> actually execute a tool
+web_search                 -> internet facts
+```
+
+Categories:
+- **recon** (10) — nmap, masscan, rustscan, netdiscover, arp-scan, amass, sublist3r,
+  theHarvester, dnsrecon, whatweb
+- **web** (7) — nikto, gobuster, dirb, ffuf, wpscan, sqlmap, commix
+- **exploit** (3) — msfconsole, searchsploit, msfvenom
+- **post** (10) — impacket-*, crackmapexec, evil-winrm, mimikatz, bloodhound-python, responder
+- **crack** (4) — hashcat, john, hydra, medusa
+- **wireless** (3) — aircrack-ng, airmon-ng, bettercap
+- **sniff** (4) — tshark, tcpdump, mitmproxy, ettercap
+- **osint** (4) — theHarvester, sherlock, holehe, spiderfoot
+- **forensics** (4) — volatility, binwalk, exiftool, strings
+- **utils** (6) — nc, socat, proxychains, curl, certutil, powershell
+
+## Controls
+| Input | Action |
+|-------|--------|
+| `exit` / `quit` | leave |
+| `!<command>` | run a shell command directly |
+| `?<query>` | search the internet (SearXNG) directly |
+| `?fetch <url>` | read a page's text |
+| `/tools` | list tool categories |
+| `/tools <cat>` | list tools in a category |
+| `/save` | save the session to markdown |
+| `/help` | help |
 
 ## Requirements
 - Python 3.11 (stdlib only — `urllib`, `subprocess`, `json`). No pip deps.
@@ -23,20 +63,15 @@ setx PATH "%PATH%;C:\Tools\metatron"
 ```
 Then `metatron` from any shell.
 
-## Controls
-| Input | Action |
-|-------|--------|
-| `exit` / `quit` | leave |
-| `!<command>` | run a shell command directly |
-| `?<query>` | search the internet (SearXNG) directly |
-
 ## Hardcoded
 - Ollama URL: `http://10.30.20.29:11434`
 - SearXNG URL: `http://10.30.20.35:6969` (self-hosted meta-search, CT 516)
-- System prompt + `run_command` / `web_search` tool definitions
+- System prompt, tool catalog, and `run_command` / `web_search` / `list_categories` /
+  `list_tools` tool definitions
 - Only the model list is dynamic (fetched from `/api/tags`).
 
-## Search notes
-- `?<query>` is the guaranteed path (no LLM needed).
-- Native `web_search` tool-calling can degenerate on nightmare `.29`'s qwen3.8fast —
-  the `?` escape always works.
+## Notes
+- `?<query>` is the guaranteed search path (no LLM needed).
+- Native tool-calling can degenerate on nightmare `.29` under GPU load; the manual
+  escapes (`!`, `?`, `/tools`) always work.
+- `keep_alive: "30m"` keeps the model resident to avoid reload latency.

metatron.py

@@ -20,16 +20,26 @@ import urllib.parse
 import urllib.request
 from html import unescape
 
+# Windows console is cp1252 — replace unencodable chars instead of crashing on
+# arbitrary Unicode the model emits (arrows, emoji, etc.).
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(errors="replace")
+    except Exception:
+        pass
+
 OLLAMA = "http://10.30.20.29:11434"   # nightmare
 SEARXNG = "http://10.30.20.35:6969"   # self-hosted meta-search (CT 516)
 HERE = os.path.dirname(os.path.abspath(__file__))
 LAST_MODEL = os.path.join(HERE, ".last_model")
 
 SYSTEM_PROMPT = (
-    "You are METATRON, a system-access AI harness running on a Windows Commando "
-    "VM. You have full system access through the run_command tool — execute shell "
-    "commands when asked and report their output. Use web_search for any up-to-date "
-    "or factual internet information. Be concise and direct."
+    "You are METATRON, a system-access AI harness on a Windows Commando VM with a "
+    "large tool catalog organized by category (recon, web, exploit, post, crack, "
+    "wireless, sniff, osint, forensics, utils). DISCOVER LAZILY: call list_categories "
+    "first, then list_tools(<category>) to see a category's tools, then run_command to "
+    "execute one. Never enumerate everything at once — drill in only as needed. Use "
+    "web_search for internet facts. Be concise and direct."
 )
 
 TOOLS = [{
@@ -60,11 +70,185 @@ TOOLS = [{
             "required": ["query"],
         },
     },
+}, {
+    "type": "function",
+    "function": {
+        "name": "list_categories",
+        "description": "List the available tool categories (recon, web, exploit, post, crack, "
+                       "wireless, sniff, osint, forensics, utils) with descriptions. Use this "
+                       "first to discover what tools exist.",
+        "parameters": {"type": "object", "properties": {}},
+    },
+}, {
+    "type": "function",
+    "function": {
+        "name": "list_tools",
+        "description": "List the tools within a category (name, description, usage). Use after "
+                       "list_categories to drill into a category.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "category": {"type": "string", "description": "Category name (e.g. 'recon', 'post')"},
+            },
+            "required": ["category"],
+        },
+    },
 }]
 
 C = {"r": "\033[0m", "b": "\033[1m", "c": "\033[36m", "g": "\033[32m",
      "y": "\033[33m", "m": "\033[35m"}
 
+# ---- Commando tool catalog, organized by category (lazy discovery) ----
+# The model sees list_categories -> list_tools -> run_command, never the whole tree.
+CATALOG = {
+    "recon": {"desc": "Network recon & scanning", "tools": [
+        {"name": "nmap", "desc": "Port/service/OS scanner",
+         "usage": "nmap -sV -sC -p- <target>"},
+        {"name": "masscan", "desc": "Ultra-fast port scanner",
+         "usage": "masscan -p1-65535 <target> --rate=1000"},
+        {"name": "rustscan", "desc": "Fast port scanner feeding nmap",
+         "usage": "rustscan -a <target>"},
+        {"name": "netdiscover", "desc": "ARP-based LAN discovery",
+         "usage": "netdiscover -r <subnet>"},
+        {"name": "arp-scan", "desc": "ARP host discovery",
+         "usage": "arp-scan -l"},
+        {"name": "amass", "desc": "Subdomain enumeration",
+         "usage": "amass enum -d <domain>"},
+        {"name": "sublist3r", "desc": "Subdomain enumeration",
+         "usage": "sublist3r -d <domain>"},
+        {"name": "theHarvester", "desc": "Email/domain OSINT",
+         "usage": "theHarvester -d <domain> -b all"},
+        {"name": "dnsrecon", "desc": "DNS reconnaissance",
+         "usage": "dnsrecon -d <domain>"},
+        {"name": "whatweb", "desc": "Web tech fingerprinting",
+         "usage": "whatweb <url>"},
+    ]},
+    "web": {"desc": "Web app testing", "tools": [
+        {"name": "nikto", "desc": "Web server vuln scanner",
+         "usage": "nikto -h <url>"},
+        {"name": "gobuster", "desc": "Directory/file brute force",
+         "usage": "gobuster dir -u <url> -w <wordlist>"},
+        {"name": "dirb", "desc": "Directory brute force",
+         "usage": "dirb <url>"},
+        {"name": "ffuf", "desc": "Fast web fuzzer",
+         "usage": "ffuf -u <url>/FUZZ -w <wordlist>"},
+        {"name": "wpscan", "desc": "WordPress scanner",
+         "usage": "wpscan --url <url>"},
+        {"name": "sqlmap", "desc": "SQL injection",
+         "usage": "sqlmap -u <url> --dbs"},
+        {"name": "commix", "desc": "Command injection",
+         "usage": "commix --url <url>"},
+    ]},
+    "exploit": {"desc": "Exploitation frameworks", "tools": [
+        {"name": "msfconsole", "desc": "Metasploit framework",
+         "usage": "msfconsole"},
+        {"name": "searchsploit", "desc": "Exploit-DB search",
+         "usage": "searchsploit <term>"},
+        {"name": "msfvenom", "desc": "Payload generation",
+         "usage": "msfvenom -p <payload> LHOST=<ip> LPORT=<port> -f exe"},
+    ]},
+    "post": {"desc": "Post-exploitation & lateral movement", "tools": [
+        {"name": "impacket-secretsdump", "desc": "Remote credential dump",
+         "usage": "impacket-secretsdump <domain>/<user>:<pass>@<target>"},
+        {"name": "impacket-psexec", "desc": "Remote shell (SMB)",
+         "usage": "impacket-psexec <domain>/<user>:<pass>@<target>"},
+        {"name": "impacket-wmiexec", "desc": "Remote shell (WMI)",
+         "usage": "impacket-wmiexec <domain>/<user>:<pass>@<target>"},
+        {"name": "impacket-GetNPUsers", "desc": "AS-REP roast (no creds)",
+         "usage": "impacket-GetNPUsers <domain>/ -usersfile <file> -no-pass"},
+        {"name": "impacket-GetUserSPNs", "desc": "Kerberoast",
+         "usage": "impacket-GetUserSPNs <domain>/<user>:<pass>"},
+        {"name": "crackmapexec", "desc": "SMB/WinRM brute & exec (also nxc)",
+         "usage": "crackmapexec smb <target>"},
+        {"name": "evil-winrm", "desc": "WinRM shell",
+         "usage": "evil-winrm -i <target> -u <user> -p <pass>"},
+        {"name": "mimikatz", "desc": "Credential dumping",
+         "usage": "mimikatz"},
+        {"name": "bloodhound-python", "desc": "AD enumeration",
+         "usage": "bloodhound-python -d <domain> -u <user> -p <pass> -ns <dc>"},
+        {"name": "responder", "desc": "LLMNR/NBT-NS poisoning",
+         "usage": "responder -I <interface>"},
+    ]},
+    "crack": {"desc": "Password cracking & brute force", "tools": [
+        {"name": "hashcat", "desc": "GPU password cracking",
+         "usage": "hashcat -m <mode> <hash> <wordlist>"},
+        {"name": "john", "desc": "Password cracking",
+         "usage": "john --wordlist=<wordlist> <hashfile>"},
+        {"name": "hydra", "desc": "Network login brute force",
+         "usage": "hydra -l <user> -P <wordlist> <target> <service>"},
+        {"name": "medusa", "desc": "Parallel brute force",
+         "usage": "medusa -h <target> -u <user> -P <wordlist> -M <module>"},
+    ]},
+    "wireless": {"desc": "Wireless attacks", "tools": [
+        {"name": "aircrack-ng", "desc": "WEP/WPA cracking",
+         "usage": "aircrack-ng -w <wordlist> <capture.cap>"},
+        {"name": "airmon-ng", "desc": "Monitor mode",
+         "usage": "airmon-ng start <interface>"},
+        {"name": "bettercap", "desc": "MITM/recon framework",
+         "usage": "bettercap"},
+    ]},
+    "sniff": {"desc": "Sniffing & MITM", "tools": [
+        {"name": "tshark", "desc": "CLI packet analysis",
+         "usage": "tshark -i <interface>"},
+        {"name": "tcpdump", "desc": "Packet capture",
+         "usage": "tcpdump -i <interface> -w out.pcap"},
+        {"name": "mitmproxy", "desc": "HTTP MITM proxy",
+         "usage": "mitmproxy"},
+        {"name": "ettercap", "desc": "MITM attacks",
+         "usage": "ettercap -T -M arp"},
+    ]},
+    "osint": {"desc": "Open-source intelligence", "tools": [
+        {"name": "theHarvester", "desc": "Email/domain harvesting",
+         "usage": "theHarvester -d <domain> -b all"},
+        {"name": "sherlock", "desc": "Username search across sites",
+         "usage": "sherlock <username>"},
+        {"name": "holehe", "desc": "Check email against services",
+         "usage": "holehe <email>"},
+        {"name": "spiderfoot", "desc": "OSINT automation",
+         "usage": "spiderfoot -s <target>"},
+    ]},
+    "forensics": {"desc": "Forensics & analysis", "tools": [
+        {"name": "volatility", "desc": "Memory forensics",
+         "usage": "volatility -f <dump> <plugin>"},
+        {"name": "binwalk", "desc": "Firmware/embedded analysis",
+         "usage": "binwalk <file>"},
+        {"name": "exiftool", "desc": "Metadata extraction",
+         "usage": "exiftool <file>"},
+        {"name": "strings", "desc": "String extraction",
+         "usage": "strings <file>"},
+    ]},
+    "utils": {"desc": "Utilities & networking", "tools": [
+        {"name": "nc", "desc": "Netcat — raw TCP/UDP",
+         "usage": "nc -lvnp <port>"},
+        {"name": "socat", "desc": "Networking relay",
+         "usage": "socat TCP-LISTEN:<port>,reuseaddr,fork TCP:<host>:<port>"},
+        {"name": "proxychains", "desc": "Proxy chaining",
+         "usage": "proxychains <cmd>"},
+        {"name": "curl", "desc": "HTTP client",
+         "usage": "curl <url>"},
+        {"name": "certutil", "desc": "Download/encode (Windows)",
+         "usage": "certutil -urlcache -f <url> <out>"},
+        {"name": "powershell", "desc": "Scripting / one-liners",
+         "usage": "powershell -c <cmd>"},
+    ]},
+}
+
+
+def list_categories():
+    lines = [f"{cat} — {info['desc']} ({len(info['tools'])} tools)"
+             for cat, info in CATALOG.items()]
+    return "\n".join(lines)
+
+
+def list_tools(category):
+    info = CATALOG.get(category)
+    if not info:
+        return (f"unknown category '{category}'. "
+                f"Categories: {', '.join(CATALOG.keys())}")
+    return "\n".join(
+        f"- {t['name']}: {t['desc']}\n    {t['usage']}"
+        for t in info["tools"])
+
 
 def get(path):
     req = urllib.request.Request(OLLAMA + path)
@@ -215,6 +399,8 @@ def print_help():
     {C['c']}!<cmd>{C['r']}        run a system command
     {C['c']}?<query>{C['r']}      search the web (SearXNG)
     {C['c']}?fetch <url>{C['r']}  read a page's text
+    {C['c']}/tools{C['r']}        list tool categories
+    {C['c']}/tools <cat>{C['r']}  list tools in a category
     {C['c']}/save{C['r']}         save this session to markdown
     {C['c']}/help{C['r']}         this list
     {C['c']}exit{C['r']}          quit
@@ -244,7 +430,7 @@ def chat(model):
     messages = [{"role": "system", "content": SYSTEM_PROMPT}]
     use_tools = supports_tools(model)
     print(f"\n{C['b']}  METATRON {C['c']}:: {model}{C['r']}  "
-          f"{C['y']}(!cmd, ?search, ?fetch, /save, /help, exit){C['r']}\n")
+          f"{C['y']}(!cmd, ?search, ?fetch, /tools, /save, exit){C['r']}\n")
     while True:
         try:
             user = input(f"{C['g']}you>{C['r']} ").strip()
@@ -261,6 +447,13 @@ def chat(model):
             fn = save_transcript(messages, model)
             print(f"  {C['g']}saved: {fn}{C['r']}\n")
             continue
+        if user == "/tools":
+            print(f"{C['c']}  [categories]{C['r']}\n  {list_categories()}\n")
+            continue
+        if user.startswith("/tools "):
+            cat = user[7:].strip()
+            print(f"{C['c']}  [tools: {cat}]{C['r']}\n  {list_tools(cat)}\n")
+            continue
         if user.startswith("!"):
             cmd = user[1:].strip()
             print(f"{C['y']}  $ {cmd}{C['r']}")
@@ -317,10 +510,14 @@ def chat(model):
                 for tc in tool_calls:
                     fn = tc.get("function", {})
                     name = fn.get("name")
+                    raw = fn.get("arguments") or {}
+                    if isinstance(raw, str):
                         try:
-                        args = json.loads(fn.get("arguments", "{}"))
+                            args = json.loads(raw)
                         except Exception:
                             args = {}
+                    else:
+                        args = raw
                     if name == "run_command":
                         cmd = args.get("command", "")
                         print(f"\n{C['y']}  $ {cmd}{C['r']}")
@@ -333,6 +530,17 @@ def chat(model):
                         out = web_search(q)
                         print(f"  {out[:600]}")
                         messages.append({"role": "tool", "content": out})
+                    elif name == "list_categories":
+                        out = list_categories()
+                        print(f"\n{C['c']}  [categories]{C['r']}")
+                        print(f"  {out}")
+                        messages.append({"role": "tool", "content": out})
+                    elif name == "list_tools":
+                        cat = args.get("category", "")
+                        out = list_tools(cat)
+                        print(f"\n{C['c']}  [tools: {cat}]{C['r']}")
+                        print(f"  {out[:900]}")
+                        messages.append({"role": "tool", "content": out})
                 continue
             else:
                 messages.append({"role": "assistant", "content": buf})