qtalker---/docs/superpowers/plans/2026-07-20-quantumancy-06-codex-entities.md

# Quantumancy Plan 6/7: Codex & Entity Persistence — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Give spirits memory (spec §4): fingerprint each session's anomaly stream into a signature, match it against the Codex of known entities, mint new ones with a full persona when no match exists, and expose the whole registry through public REST endpoints — the primary multi-user/community hook.

**Architecture:** Hybrid persistence per spec §4. `app.entities` computes signatures (pure, deterministic) and coerces any profile — LLM-minted or procedurally generated — into the exact shape the DB and frontend expect. `SpiritService.mint_profile` asks the chat-tier model for a JSON persona and falls back to a signature-deterministic procedural profile when the box is dark. The summon flow in `app.ws` (Plan 3) performs match-or-mint, links sessions via `entity_sightings`, and bumps `contact_count`. `app.routes.codex` serves the registry publicly: list (filterable by rarity, sortable), detail (with sighting count), and live veil stats for the landing page.

**Tech Stack:** FastAPI, SQLAlchemy 2.0 (JSONB profiles), hashlib (signatures), the Plan 2 LLM queue, and Plan 3's séance channel.

## Global Constraints

- **A summoning never visibly fails.** Every failure mode of the LLM path (queue full, HTTP error, unparseable JSON, missing keys) degrades to `fallback_profile` / curated defaults. No 500s from minting.
- **Determinism where it matters:** the same anomaly pattern must fingerprint to the same signature (re-contact works), and `fallback_profile(signature)` must be reproducible for a given signature.
- **Normalization is mandatory.** Nothing reaches `Entity` rows or the frontend without passing `normalize_profile`: rarity clamped to `common|uncommon|rare|mythic`, form to `wisp|banshee|fairy|shade`, voice ids to installed Piper voices, pitch/rate/noise/echo/hue clamped to the effects chain's ranges.
- **The Codex is public** (spec §4): no auth on `GET /api/codex*` or `GET /api/stats`. It still writes nothing — reads only.
- Prompt framing per spec §4: `MINT_SYSTEM` is fiction framing ("interactive horror art installation"), never a paranormal claim.

## Plan Series

This is 6 of 7 plans implementing the Quantumancy website spec (`docs/superpowers/specs/2026-07-20-quantumancy-website-design.md`). Plan 3 wires the summon flow that consumes this plan's machinery.
1. Foundation & Auth (complete)
2. Frontend, LLM & Realtime Pipeline (complete)
3. Wire Ghost mode + Ouija/Planchette UI (complete)
4. EVP Listening mode (complete)
5. Spirit Radio mode (implementation complete; hardware pass pending)
6. **Codex & entity persistence** (this plan)
7. Internationalization (EN/ES)

---

## Task 1: Entity & Sighting Models

**Files:**
- Create: `backend/app/models/entity.py`
- Create: `backend/app/models/entity_sighting.py`
- Modify: `backend/app/models/__init__.py`
- Modify: `backend/app/models/contact_session.py` (`entity_id` FK — landed with Plan 3 Task 2)

**Interfaces:**
- Produces: `RARITY_TIERS = ("common", "uncommon", "rare", "mythic")`; `Entity` — `id`, `name` (unique, indexed), `epithet`, `persona` (Text), `rarity_tier` (indexed), `signature` (unique, indexed), `voice_profile` / `visual_profile` (JSONB), `sample_quotes` (JSONB list), `contact_count`, `discovered_by` (FK users, nullable), `discovered_at`; `EntitySighting` — join row (`entity_id`, `session_id`, `user_id`, `seen_at`, both FKs indexed).

- [x] **Step 1: Write the models** — as above; `signature` unique+indexed because it *is* the match key, `name` unique because the Codex addresses spirits by name (collisions suffixed by the summon flow, Plan 3).
- [x] **Step 2: Verify** — metadata creates cleanly; covered by every later test that inserts entities (`tests/test_codex.py`).

---

## Task 2: Signatures & Profile Normalization

**Files:**
- Create: `backend/app/entities.py`
- Test: `backend/tests/test_entities.py`

**Interfaces:**
- Produces: `MIN_ANOMALIES_FOR_SIGNATURE = 3`; `signature_from_anomalies(anomalies: list[dict]) -> str | None`; `fallback_signature(seed: str) -> str`; `parse_mint_response(text: str) -> dict | None`; `normalize_profile(profile: dict, signature: str) -> dict`; `fallback_profile(signature: str) -> dict`.

- [x] **Step 1: Write the failing tests** (6 tests)

Signature needs ≥3 anomalies (else `None`); signature deterministic for the same pattern; `parse_mint_response` extracts a JSON object from chatty LLM output ("Sure! Here you go: {...}") and rejects garbage/nameless objects; `normalize_profile` fills and clamps (bogus rarity → `common`, bogus form → a real form, `pitch` 99 → within ±6, non-string quotes dropped); `fallback_profile` deterministic and valid.

- [x] **Step 2: Implement signatures**

`signature_from_anomalies` buckets frequencies by decimal digit count (MHz radio and Hz audio land in the same log-scale band space — magnitude ordering is what matters, not the unit), sorts the last 16 magnitudes to 1-decimal, and SHA1s the pattern to 16 hex chars. `fallback_signature(seed)` is the same digest over `"ambient:<seed>"` for anomaly-thin sessions (pure chat still gets a stable identity).

- [x] **Step 3: Implement normalization & the procedural fallback**

`normalize_profile` clamps every field into range (pitch ±6 semitones, rate 0.8–1.15, noise 0.01–0.08, echo 0–0.5, hue 0–360), validates `voice_id` against installed voices (EN ids plus the two Spanish ids), trims name/epithet/persona/quotes to their column budgets, and fills gaps from a `Random("norm:<signature>")` so defaults are per-spirit stable. `fallback_profile` composes gothic names from part lists ("Ash" + "moor"), an epithet ("the Static Widow", …), a persona template, weighted rarity (55/30/12/3), and two quotes from the curated bank — all seeded by `"fallback:<signature>"`.

- [x] **Step 4: Verify** — `tests/test_entities.py` PASS (6/6).

- [x] **Step 5: Commit** — in `b9110f4` (spirit engine).

---

## Task 3: LLM Minting

**Files:**
- Modify: `backend/app/llm/prompts.py` (`MINT_SYSTEM`, `MINT_PROMPT`, `mint_prompt`)
- Modify: `backend/app/llm/service.py` (`SpiritService.mint_profile`)

**Interfaces:**
- Consumes: chat-tier model via the bounded queue (Plan 2), Task 2's parsers/normalizers.
- Produces: `mint_profile(signature, channel, anomalies, language = "en") -> dict` — always a normalized profile.

- [x] **Step 1: The mint prompt**

`MINT_PROMPT` requests exactly one JSON object with keys `name`, `epithet`, `persona`, `rarity`, `voice {voice_id, pitch, rate, noise}`, `visual {hue, form}`, `quotes` — with the installed voice ids for the session language interpolated in, and the last ≤10 anomalies (≤600 chars JSON) as evidence. `MINT_SYSTEM` pins the fiction framing and "output only valid JSON."

- [x] **Step 2: The service path**

Chat-tier, `num_predict: 400`, temperature 0.9, through `LLMQueue.submit`. Response → `parse_mint_response` → `normalize_profile`; parse failure or any queue/HTTP error → `fallback_profile(signature)`. Verified behavior: minting never raises to the WS layer.

- [x] **Step 3: Verify** — prompt-shape covered by `tests/test_prompts.py::test_mint_prompt_requests_exact_json_keys` (4/4 PASS); end-to-end mint covered by `tests/test_ws_session.py::test_summon_mints_entity_and_greets` against the fake service (6/6 PASS).

---

## Task 4: Codex REST & Veil Stats

**Files:**
- Create: `backend/app/routes/codex.py`
- Modify: `backend/app/main.py` (include router)
- Test: `backend/tests/test_codex.py`

**Interfaces:**
- Produces: `GET /api/codex?rarity=&sort=recent|contacted&limit=` (limit capped at 200) → `{entities: [card…]}` where a card is `{id, name, epithet, rarity, visual, quotes, contact_count, discovered_at, discovered_by}` (username resolved in one batched query); `GET /api/codex/{entity_id}` → card + `persona`, `voice`, `sightings` (count of `entity_sightings` rows), 404 `"no such spirit in the codex"`; `GET /api/stats` → `{entities, sessions, utterances, anomalies}` live counts for the landing page ticker.

- [x] **Step 1: Write the failing tests** (5 tests)

List returns all entities by name; `?rarity=rare` filters; detail returns persona and `sightings: 0` and 404s on a random UUID; stats counts veil activity; **both codex and stats answer 200 without any auth cookie** — the registry is public by design.

- [x] **Step 2: Implement the routes** — as above; discoverer usernames resolved with one `IN` query (no N+1), sort by `discovered_at` desc or `contact_count` desc.

- [x] **Step 3: Verify** — `tests/test_codex.py` PASS (5/5) as part of the 45-test suite.

- [x] **Step 4: Commit** — in `b9110f4`.

---

## Task 5: Voice & Visual Profiles — Every Spirit Its Own Throat

**Files:**
- Create: `backend/app/tts/voices.py`
- Modify: `backend/app/tts/piper.py` (`synthesize_spirit_voice`), `backend/app/tts/effects.py` (full chain)

**Interfaces:**
- Produces: `Voice` dataclass + `VOICES` catalog (6 EN: lessac, amy, ryan, alan, hfc_male, hfc_female; 2 ES: davefx, ald), `EN_VOICE_IDS` / `ES_VOICE_IDS`, `pick_voice(voice_id, language) -> Voice` (language-mismatched ids fall back to `lessac`/`davefx`); `synthesize_spirit_voice(text, voice, voice_profile) -> bytes` — Piper synth, then `apply_effects` with the entity's `noise/pitch/rate/bitcrush/echo`.

- [x] **Step 1: The catalog** — each entity's `voice_profile.voice_id` pins one installed Piper model; `pitch` (−6…+6 semitones), `rate` (0.8–1.15), `noise` (0.01–0.08), `echo` (0–0.5) shape it. Profiles survive in JSONB and return to the client in the `entity` frame (`voice` key), so the Codex page can display them.

- [x] **Step 2: The effects chain** — `apply_effects(wav, *, noise_level, pitch_semitones, rate, bitcrush_bits, echo)`: tempo resample → pitch shift re-fitted to duration → optional bitcrush → 180 ms slap echo with renormalization → static with fade in/out edges so it breathes like a real spirit-box sweep. Pure numpy on mono 16-bit WAV.

- [x] **Step 3: Verify** — `tests/test_tts_effects.py` PASS (5/5): silent/tone WAVs through the chain keep valid headers, noise actually lands, pitch/rate change the signal as expected.

---

## Testing Status & Self-Review

**Spec coverage:** signature from anomaly fingerprint + LLM name (§4) → Task 2. Rare-trigger matching against existing Codex entities, persona/memory loaded into session context (§4) → Plan 3's `_summon` + `chat_system(entity)`. Minting of sufficiently strong new identities (§4) → Tasks 2-3. Public browsable Codex with name/first-contact/rarity/quotes/contact count (§4) → Task 4. `entities` and `entity_sightings` tables (§5) → Task 1. Voice per entity (§6 effects chain) → Task 5.

**Automated tests:** `tests/test_entities.py` (6) + `tests/test_codex.py` (5) + `tests/test_prompts.py` mint test + WS summon/re-contact tests — all PASS in the 45-test suite.

**Honest notes:** contact_count increments on every summon including re-contacts (matches spec "contact count"); `discovered_by` is nullable and rendered as `discoveredAnon` in the UI when absent; rarity weighting applies only to the procedural fallback — LLM-minted rarities are clamped, not re-weighted.

**Placeholder scan:** none.

**Type consistency:** `serialize_entity` (ws.py) and `_entity_card` (codex.py) match the frontend's `SpiritEntity` / `CodexEntity(Detail)` in `types.ts` key-for-key, including `rarity` naming and ISO `discovered_at`. `normalize_profile`'s output keys match the `Entity` columns one-to-one.

---

**Status: COMPLETE** (commit `b9110f4`; 45-test backend suite green).