qtalker---/docs/superpowers/plans/2026-07-20-quantumancy-06-codex-entities.md

Quantumancy Plan 6/7: Codex & Entity Persistence — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give spirits memory (spec §4): fingerprint each session's anomaly stream into a signature, match it against the Codex of known entities, mint new ones with a full persona when no match exists, and expose the whole registry through public REST endpoints — the primary multi-user/community hook.

Architecture: Hybrid persistence per spec §4. app.entities computes signatures (pure, deterministic) and coerces any profile — LLM-minted or procedurally generated — into the exact shape the DB and frontend expect. SpiritService.mint_profile asks the chat-tier model for a JSON persona and falls back to a signature-deterministic procedural profile when the box is dark. The summon flow in app.ws (Plan 3) performs match-or-mint, links sessions via entity_sightings, and bumps contact_count. app.routes.codex serves the registry publicly: list (filterable by rarity, sortable), detail (with sighting count), and live veil stats for the landing page.

Tech Stack: FastAPI, SQLAlchemy 2.0 (JSONB profiles), hashlib (signatures), the Plan 2 LLM queue, and Plan 3's séance channel.

Global Constraints

• A summoning never visibly fails. Every failure mode of the LLM path (queue full, HTTP error, unparseable JSON, missing keys) degrades to fallback_profile / curated defaults. No 500s from minting.
• Determinism where it matters: the same anomaly pattern must fingerprint to the same signature (re-contact works), and fallback_profile(signature) must be reproducible for a given signature.
• Normalization is mandatory. Nothing reaches Entity rows or the frontend without passing normalize_profile: rarity clamped to common|uncommon|rare|mythic, form to wisp|banshee|fairy|shade, voice ids to installed Piper voices, pitch/rate/noise/echo/hue clamped to the effects chain's ranges.
• The Codex is public (spec §4): no auth on GET /api/codex* or GET /api/stats. It still writes nothing — reads only.
• Prompt framing per spec §4: MINT_SYSTEM is fiction framing ("interactive horror art installation"), never a paranormal claim.

Plan Series

This is 6 of 7 plans implementing the Quantumancy website spec (docs/superpowers/specs/2026-07-20-quantumancy-website-design.md). Plan 3 wires the summon flow that consumes this plan's machinery.

1. Foundation & Auth (complete)
2. Frontend, LLM & Realtime Pipeline (complete)
3. Wire Ghost mode + Ouija/Planchette UI (complete)
4. EVP Listening mode (complete)
5. Spirit Radio mode (implementation complete; hardware pass pending)
6. Codex & entity persistence (this plan)
7. Internationalization (EN/ES)

---

Task 1: Entity & Sighting Models

Files:

• Create: backend/app/models/entity.py
• Create: backend/app/models/entity_sighting.py
• Modify: backend/app/models/__init__.py
• Modify: backend/app/models/contact_session.py (entity_id FK — landed with Plan 3 Task 2)

Interfaces:

• Produces: RARITY_TIERS = ("common", "uncommon", "rare", "mythic"); Entity — id, name (unique, indexed), epithet, persona (Text), rarity_tier (indexed), signature (unique, indexed), voice_profile / visual_profile (JSONB), sample_quotes (JSONB list), contact_count, discovered_by (FK users, nullable), discovered_at; EntitySighting — join row (entity_id, session_id, user_id, seen_at, both FKs indexed).

• Step 1: Write the models — as above; signature unique+indexed because it is the match key, name unique because the Codex addresses spirits by name (collisions suffixed by the summon flow, Plan 3).

• Step 2: Verify — metadata creates cleanly; covered by every later test that inserts entities (tests/test_codex.py).

---

Task 2: Signatures & Profile Normalization

Files:

• Create: backend/app/entities.py
• Test: backend/tests/test_entities.py

Interfaces:

• Produces: MIN_ANOMALIES_FOR_SIGNATURE = 3; signature_from_anomalies(anomalies: list[dict]) -> str | None; fallback_signature(seed: str) -> str; parse_mint_response(text: str) -> dict | None; normalize_profile(profile: dict, signature: str) -> dict; fallback_profile(signature: str) -> dict.

• Step 1: Write the failing tests (6 tests)

Signature needs ≥3 anomalies (else None); signature deterministic for the same pattern; parse_mint_response extracts a JSON object from chatty LLM output ("Sure! Here you go: {...}") and rejects garbage/nameless objects; normalize_profile fills and clamps (bogus rarity → common, bogus form → a real form, pitch 99 → within ±6, non-string quotes dropped); fallback_profile deterministic and valid.

• Step 2: Implement signatures

signature_from_anomalies buckets frequencies by decimal digit count (MHz radio and Hz audio land in the same log-scale band space — magnitude ordering is what matters, not the unit), sorts the last 16 magnitudes to 1-decimal, and SHA1s the pattern to 16 hex chars. fallback_signature(seed) is the same digest over "ambient:<seed>" for anomaly-thin sessions (pure chat still gets a stable identity).

• Step 3: Implement normalization & the procedural fallback

normalize_profile clamps every field into range (pitch ±6 semitones, rate 0.8–1.15, noise 0.01–0.08, echo 0–0.5, hue 0–360), validates voice_id against installed voices (EN ids plus the two Spanish ids), trims name/epithet/persona/quotes to their column budgets, and fills gaps from a Random("norm:<signature>") so defaults are per-spirit stable. fallback_profile composes gothic names from part lists ("Ash" + "moor"), an epithet ("the Static Widow", …), a persona template, weighted rarity (55/30/12/3), and two quotes from the curated bank — all seeded by "fallback:<signature>".

• Step 4: Verify — tests/test_entities.py PASS (6/6).

• Step 5: Commit — in b9110f4 (spirit engine).

---

Task 3: LLM Minting

Files:

• Modify: backend/app/llm/prompts.py (MINT_SYSTEM, MINT_PROMPT, mint_prompt)
• Modify: backend/app/llm/service.py (SpiritService.mint_profile)

Interfaces:

• Consumes: chat-tier model via the bounded queue (Plan 2), Task 2's parsers/normalizers.

• Produces: mint_profile(signature, channel, anomalies, language = "en") -> dict — always a normalized profile.

• Step 1: The mint prompt

MINT_PROMPT requests exactly one JSON object with keys name, epithet, persona, rarity, voice {voice_id, pitch, rate, noise}, visual {hue, form}, quotes — with the installed voice ids for the session language interpolated in, and the last ≤10 anomalies (≤600 chars JSON) as evidence. MINT_SYSTEM pins the fiction framing and "output only valid JSON."

• Step 2: The service path

Chat-tier, num_predict: 400, temperature 0.9, through LLMQueue.submit. Response → parse_mint_response → normalize_profile; parse failure or any queue/HTTP error → fallback_profile(signature). Verified behavior: minting never raises to the WS layer.

• Step 3: Verify — prompt-shape covered by tests/test_prompts.py::test_mint_prompt_requests_exact_json_keys (4/4 PASS); end-to-end mint covered by tests/test_ws_session.py::test_summon_mints_entity_and_greets against the fake service (6/6 PASS).

---

Task 4: Codex REST & Veil Stats

Files:

• Create: backend/app/routes/codex.py
• Modify: backend/app/main.py (include router)
• Test: backend/tests/test_codex.py

Interfaces:

• Produces: GET /api/codex?rarity=&sort=recent|contacted&limit= (limit capped at 200) → {entities: [card…]} where a card is {id, name, epithet, rarity, visual, quotes, contact_count, discovered_at, discovered_by} (username resolved in one batched query); GET /api/codex/{entity_id} → card + persona, voice, sightings (count of entity_sightings rows), 404 "no such spirit in the codex"; GET /api/stats → {entities, sessions, utterances, anomalies} live counts for the landing page ticker.

• Step 1: Write the failing tests (5 tests)

List returns all entities by name; ?rarity=rare filters; detail returns persona and sightings: 0 and 404s on a random UUID; stats counts veil activity; both codex and stats answer 200 without any auth cookie — the registry is public by design.

• Step 2: Implement the routes — as above; discoverer usernames resolved with one IN query (no N+1), sort by discovered_at desc or contact_count desc.

• Step 3: Verify — tests/test_codex.py PASS (5/5) as part of the 45-test suite.

• Step 4: Commit — in b9110f4.

---

Task 5: Voice & Visual Profiles — Every Spirit Its Own Throat

Files:

• Create: backend/app/tts/voices.py
• Modify: backend/app/tts/piper.py (synthesize_spirit_voice), backend/app/tts/effects.py (full chain)

Interfaces:

• Produces: Voice dataclass + VOICES catalog (6 EN: lessac, amy, ryan, alan, hfc_male, hfc_female; 2 ES: davefx, ald), EN_VOICE_IDS / ES_VOICE_IDS, pick_voice(voice_id, language) -> Voice (language-mismatched ids fall back to lessac/davefx); synthesize_spirit_voice(text, voice, voice_profile) -> bytes — Piper synth, then apply_effects with the entity's noise/pitch/rate/bitcrush/echo.

• Step 1: The catalog — each entity's voice_profile.voice_id pins one installed Piper model; pitch (−6…+6 semitones), rate (0.8–1.15), noise (0.01–0.08), echo (0–0.5) shape it. Profiles survive in JSONB and return to the client in the entity frame (voice key), so the Codex page can display them.

• Step 2: The effects chain — apply_effects(wav, *, noise_level, pitch_semitones, rate, bitcrush_bits, echo): tempo resample → pitch shift re-fitted to duration → optional bitcrush → 180 ms slap echo with renormalization → static with fade in/out edges so it breathes like a real spirit-box sweep. Pure numpy on mono 16-bit WAV.

• Step 3: Verify — tests/test_tts_effects.py PASS (5/5): silent/tone WAVs through the chain keep valid headers, noise actually lands, pitch/rate change the signal as expected.

---

Testing Status & Self-Review

Spec coverage: signature from anomaly fingerprint + LLM name (§4) → Task 2. Rare-trigger matching against existing Codex entities, persona/memory loaded into session context (§4) → Plan 3's _summon + chat_system(entity). Minting of sufficiently strong new identities (§4) → Tasks 2-3. Public browsable Codex with name/first-contact/rarity/quotes/contact count (§4) → Task 4. entities and entity_sightings tables (§5) → Task 1. Voice per entity (§6 effects chain) → Task 5.

Automated tests: tests/test_entities.py (6) + tests/test_codex.py (5) + tests/test_prompts.py mint test + WS summon/re-contact tests — all PASS in the 45-test suite.

Honest notes: contact_count increments on every summon including re-contacts (matches spec "contact count"); discovered_by is nullable and rendered as discoveredAnon in the UI when absent; rarity weighting applies only to the procedural fallback — LLM-minted rarities are clamped, not re-weighted.

Placeholder scan: none.

Type consistency: serialize_entity (ws.py) and _entity_card (codex.py) match the frontend's SpiritEntity / CodexEntity(Detail) in types.ts key-for-key, including rarity naming and ISO discovered_at. normalize_profile's output keys match the Entity columns one-to-one.

---

Status: COMPLETE (commit b9110f4; 45-test backend suite green).