feat(firmware): add experimental RTL-SDR USB-host module (Workstream J)
Self-contained ESP-IDF component (firmware/esp32p4-sensor-node/components/ rtlsdr_experimental/) exploring RTL2832U-over-USB-host on the ESP32-P4, ported from frontend/src/lib/sdr.ts's researched WebUSB protocol sequence (vendor commands, I2C-repeater tuner init) to the ESP-IDF USB Host Library. Implements: USB Host Library install/client lifecycle, RTL2832U/Terratec vendor-ID device matching, the demod+R820T init vendor-command sequence over control transfers, a pipelined bulk-IN read loop for raw IQ, and an inert-by-default upstream IQ-forwarding stub targeting a proposed separate binary endpoint (not the JSON telemetry shape — reasoning documented in the README) since no such backend endpoint exists yet. Off by default (RTLSDR_EXP_ENABLE Kconfig, default n). Unverified against real hardware and never compiled (no ESP-IDF toolchain in this environment) — marked as such in every source file and in a dedicated "Workstream J" section of firmware/esp32p4-sensor-node/README.md, which this commit also creates since Workstream I's core skeleton (owned by a separate, unmerged worktree) hadn't created one yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Indiana
324
firmware/esp32p4-sensor-node/README.md
Normal file
324
firmware/esp32p4-sensor-node/README.md
Normal file
@@ -0,0 +1,324 @@
|
||||
# ESP32-P4 Sensor Node — firmware
|
||||
|
||||
Firmware for the physical ESP32-P4 "Ultimate Quantum Box" sensor node
|
||||
(see the ESP32-P4 Sensor Node design spec:
|
||||
`docs/superpowers/specs/2026-07-23-esp32-sensor-node-design.md`). Joins
|
||||
the seeker's home WiFi and streams sensor readings to `POST
|
||||
/api/device/telemetry` on their paired account, which feeds the séance's
|
||||
live anomaly-detection pipeline exactly like the browser-based Wire
|
||||
Ghost/EVP/Spirit Radio/EMF modes already do.
|
||||
|
||||
## Honesty-policy note (binding on this whole directory)
|
||||
|
||||
This app's whole ethos is "real signal processing on real data, and it
|
||||
says so when something is unverified" (see the frontend's Spirit Radio
|
||||
`HARDWARE PASS REQUIRED` convention in `frontend/src/lib/sdr.ts`). Nobody
|
||||
working on this firmware has physical ESP32-P4 hardware to flash and test
|
||||
against, and this development environment has no ESP-IDF toolchain to
|
||||
even compile it. **Every file under this directory is written to be
|
||||
structurally correct ESP-IDF C against the documented API shape, and
|
||||
reasoned about carefully — not verified against real hardware or even a
|
||||
real build.** Anything that needs a real board to actually confirm is
|
||||
called out explicitly, in code comments and in this README, rather than
|
||||
glossed over. Treat "compiles cleanly / structurally sound" and "verified
|
||||
on-device" as two entirely different claims — this repo only makes the
|
||||
first one for anything under `firmware/`.
|
||||
|
||||
## Status of this directory
|
||||
|
||||
This copy of the repo currently contains **only Workstream J** (the
|
||||
RTL-SDR experimental module, documented in full below). Workstream I
|
||||
— the core firmware project skeleton (`CMakeLists.txt`,
|
||||
`sdkconfig.defaults`, the `main/` component with WiFi station-mode
|
||||
connect, the `/api/device/telemetry` HTTP client task, `main/
|
||||
device_config.h`, and the BME280 + presence-sensor drivers) — was built
|
||||
in a separate, isolated worktree that this workstream cannot see or
|
||||
depend on. **A real build of this project needs Workstream I's skeleton
|
||||
merged in alongside what's here.** Nothing in this directory currently
|
||||
compiles standalone into a flashable image; the `components/
|
||||
rtlsdr_experimental/` module is a self-contained ESP-IDF *component*,
|
||||
deliberately structured so it can be dropped into Workstream I's project
|
||||
via `EXTRA_COMPONENT_DIRS` (or copied under that project's own
|
||||
`components/`) without editing any file that workstream owns. Once
|
||||
merged, this README should be extended with Workstream I's own build
|
||||
steps, wiring/pinout notes for the BME280 (I2C) and presence sensor
|
||||
(UART/GPIO), and its own verified-vs-not accounting — that content
|
||||
doesn't exist here yet because this workstream never had visibility into
|
||||
that code.
|
||||
|
||||
## Directory layout (this workstream's contribution)
|
||||
|
||||
```
|
||||
firmware/esp32p4-sensor-node/
|
||||
├── README.md (this file)
|
||||
└── components/
|
||||
└── rtlsdr_experimental/ Self-contained ESP-IDF component.
|
||||
├── CMakeLists.txt
|
||||
├── Kconfig Off by default (RTLSDR_EXP_ENABLE=n).
|
||||
├── include/
|
||||
│ └── rtlsdr_experimental.h Public API + full disclaimer header.
|
||||
└── rtlsdr_experimental.c Implementation.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workstream J — RTL-SDR experimental module (this workstream)
|
||||
|
||||
**⚠️ This is the least certain part of the entire firmware build. Read
|
||||
this whole section before touching it.**
|
||||
|
||||
### What this is
|
||||
|
||||
A clearly-separated, opt-in, experimental module exploring USB-host
|
||||
communication with an RTL2832U-based SDR dongle over the ESP32-P4's
|
||||
native USB-OTG host controller (USB Host Library, `usb_host.h` — a real,
|
||||
documented capability of this specific chip, unlike most ESP32 variants).
|
||||
The idea, per the design spec: attach a cheap RTL-SDR dongle to the
|
||||
sensor node, treat "spirit radio scanning" as a hardware-backed mode
|
||||
instead of only a browser WebUSB feature.
|
||||
|
||||
**Be honest about the real constraint (this is the spec's framing, and
|
||||
it's correct):** wideband IQ sample rates and FFT processing are
|
||||
demanding relative to an MCU's compute, even one with ESP32-P4's AI
|
||||
accelerator. **On-device spectrum analysis is not what this module
|
||||
attempts.** The realistic architecture — and the one implemented here —
|
||||
is: pull raw IQ samples off the dongle via USB host, and forward them
|
||||
upstream for the *backend* to FFT/analyze (the same job the browser
|
||||
already does client-side today via `frontend/src/lib/fft.ts` +
|
||||
`frontend/src/lib/sdr.ts`'s `SpectrumAnomalyDetector`). Even that
|
||||
"just pass the bytes through" architecture needs real USB throughput
|
||||
numbers to know if it's viable — see "What's unverified" below.
|
||||
|
||||
### Primary reference
|
||||
|
||||
`frontend/src/lib/sdr.ts` — this repo's existing browser-based (WebUSB)
|
||||
RTL2832U + R820T driver. It's real, already-researched protocol detail
|
||||
against the public librtlsdr register documentation (vendor commands,
|
||||
I2C-repeater tuner access, the demod/tuner init sequence), itself marked
|
||||
`HARDWARE PASS REQUIRED` since it's never been run against a real dongle
|
||||
either. This firmware module is a direct **port** of that file's control-
|
||||
transfer sequence from WebUSB JS calls to ESP-IDF USB Host Library C
|
||||
calls — line-by-line correspondences are called out in code comments
|
||||
(e.g. `rtlsdr_run_init_sequence()` mirrors `open()` in sdr.ts almost
|
||||
register-for-register). It intentionally does not re-derive any register
|
||||
math from scratch; wherever sdr.ts says "simplified" or "HARDWARE PASS
|
||||
REQUIRED" (e.g. the R820T PLL frequency math, the demod resample-ratio
|
||||
math), this module carries the exact same simplification forward with the
|
||||
exact same caveat, rather than inventing new unverified math on top of
|
||||
already-unverified math.
|
||||
|
||||
### What's implemented (structurally — see caveats below)
|
||||
|
||||
1. **USB Host Library lifecycle** (`rtlsdr_exp_start()` /
|
||||
`rtlsdr_usb_lib_daemon_task()` / `rtlsdr_exp_client_task()`):
|
||||
`usb_host_install()`, `usb_host_client_register()` with an async event
|
||||
callback, and the two-task pump pattern the USB Host Library's async
|
||||
model requires (one task for `usb_host_lib_handle_events()`, one for
|
||||
`usb_host_client_handle_events()` — the latter also being how this
|
||||
module's own control- and bulk-transfer completion callbacks get
|
||||
dispatched, since the Library calls them synchronously from whichever
|
||||
task is pumping events, not from a hidden thread or ISR).
|
||||
2. **RTL2832U/Terratec device enumeration and vendor-ID matching**
|
||||
(`rtlsdr_try_bring_up()`, `rtlsdr_vendor_id_matches()`): on a
|
||||
`USB_HOST_CLIENT_EVENT_NEW_DEV` event, opens the device, reads its
|
||||
device descriptor, and matches `idVendor` against `0x0BDA`
|
||||
(RTL2832U) / `0x0CCD` (Terratec-rebadged) — ported directly from
|
||||
`sdr.ts`'s `requestDevice()` filter list (`RTL2832U_VENDOR`,
|
||||
`TERRATEC_VENDOR`). Faithfully carries over that file's specific
|
||||
choice to filter by **vendor ID only**, not product ID (its comment:
|
||||
"many dongles report product ids outside the handful we know, so
|
||||
filtering by productId hides them from the picker") — the known
|
||||
product-id list (`RTLSDR_EXP_KNOWN_PRODUCT_IDS`) is kept as an
|
||||
informational log line only, never a hard filter, exactly mirroring
|
||||
how `RTL2832U_PRODUCTS` is exported-but-unused-as-a-filter in
|
||||
`sdr.ts`.
|
||||
3. **RTL2832U init vendor-command sequence** over USB control transfers
|
||||
(`rtlsdr_run_init_sequence()`, `rtlsdr_demod_write()`,
|
||||
`rtlsdr_reg_write()`, `rtlsdr_i2c_write()`, `rtlsdr_exp_set_frequency()`,
|
||||
`rtlsdr_exp_set_sample_rate()`): the same demod soft-reset → demod_ctl/
|
||||
suspend-off block → standby-off → AGC-mode → R820T tuner power-up
|
||||
(through the I2C repeater) → sample-rate program → initial tune →
|
||||
streaming-endpoint reset → test-mode-off sequence as `sdr.ts`'s
|
||||
`open()`, with the same register addresses/values and the same
|
||||
`wValue`/`wIndex` encoding (`(block<<8)|0x10` / `(page<<8)|address`),
|
||||
translated from `USBDevice.controlTransferOut()` to
|
||||
`usb_host_transfer_submit_control()` with a manually-built
|
||||
`usb_setup_packet_t`.
|
||||
4. **A basic bulk-transfer read loop structure** for pulling IQ sample
|
||||
data off the device (`rtlsdr_start_bulk_streaming()`,
|
||||
`rtlsdr_bulk_xfer_cb()`): unlike `sdr.ts` (which does one-shot
|
||||
`await dev.transferIn(...)` calls from inside its own async `sweep()`
|
||||
loop), the USB Host Library is callback-driven, so continuous
|
||||
streaming here is a small pipeline — N transfers (`bulk_read_queue_
|
||||
depth`, default 4) of a fixed chunk size (`bulk_read_chunk_bytes`,
|
||||
default 16384B, enforced as a multiple of 512 the same way `sdr.ts`'s
|
||||
`readSamples()` comment requires) are kept perpetually in flight; each
|
||||
completion callback copies the received bytes into a heap block, hands
|
||||
it to a FreeRTOS queue for the forward task, and immediately resubmits
|
||||
itself to keep the pipe full. **This is the part most likely to need
|
||||
real-hardware tuning** — see below.
|
||||
5. **A stub/structure for forwarding raw IQ data upstream**
|
||||
(`rtlsdr_exp_forward_iq_block()`, `rtlsdr_forward_task()`) — see the
|
||||
architecture decision writeup immediately below.
|
||||
|
||||
### IQ-forwarding architecture decision
|
||||
|
||||
The spec explicitly leaves this as a product-judgment call ("your call on
|
||||
whether this warrants a separate endpoint/stream vs. reusing the sensor
|
||||
telemetry shape with a binary/base64 payload"). Decision made here:
|
||||
**a separate binary stream/endpoint, not a reuse of the JSON telemetry
|
||||
shape.** Reasoning:
|
||||
|
||||
- **Raw throughput is the wrong order of magnitude for the telemetry
|
||||
contract.** Even a modest 2.048 Msps capture at 8 bits/sample/channel
|
||||
(RTL2832U's native ADC format) is interleaved I/Q bytes at roughly
|
||||
4.1 MB/s. The design spec's `POST /api/device/telemetry` contract caps
|
||||
the `readings` array at ~64 entries, caps the request body at ~16KB,
|
||||
and rate-limits ingestion to roughly 1 request/second sustained per
|
||||
device — sized for a handful of scalar sensor readings (temperature,
|
||||
humidity, presence booleans), not a continuous multi-megabyte/second
|
||||
binary stream. Forcing IQ data through that shape would mean either
|
||||
violating those caps (defeating their whole purpose — bounding what a
|
||||
misbehaving/malicious device can push at the backend) or chopping IQ
|
||||
into thousands of tiny requests per second, which is worse for both
|
||||
sides than one continuous stream.
|
||||
- **Base64-in-JSON adds ~33% size overhead** on top of a payload that's
|
||||
already too large for the telemetry shape, for no benefit — there's no
|
||||
reason to pay text-encoding tax on a payload nothing needs to
|
||||
eyeball as text.
|
||||
- **It's a fundamentally different kind of data with different backend
|
||||
handling needs.** Telemetry readings feed the anomaly-detection
|
||||
baseline/threshold pipeline directly and cheaply (a few floats per
|
||||
sensor). IQ data needs FFT processing before it's useful for anything
|
||||
— an entirely different backend code path (closer to how the browser's
|
||||
own `powerSpectrumDb()` + `SpectrumAnomalyDetector` work today,
|
||||
server-side instead of client-side). Conflating the two request shapes
|
||||
would couple two things that should scale, rate-limit, and fail
|
||||
independently.
|
||||
|
||||
Given that, this module's stub (`rtlsdr_exp_forward_iq_block()`) targets
|
||||
a **separate, currently-hypothetical** endpoint (`iq_upload_url` in
|
||||
`rtlsdr_exp_config_t`, suggested path `/api/device/iq-stream` in code
|
||||
comments) carrying a small fixed binary header (magic, sequence number,
|
||||
center frequency, sample rate, payload length — see
|
||||
`rtlsdr_iq_chunk_header_t`) followed by the raw IQ bytes, POSTed as
|
||||
`application/octet-stream` with the same `Authorization: Bearer <device
|
||||
token>` auth as the regular telemetry loop. **This endpoint does not
|
||||
exist anywhere in this repo.** Building it is explicitly out of scope for
|
||||
this workstream (it's backend work, not firmware, and the spec doesn't
|
||||
assign a backend workstream to receive IQ data at all — only to receive
|
||||
scalar telemetry). The stub is inert by default (`rtlsdr_exp_forward_
|
||||
iq_block()` returns `ESP_OK` immediately unless `iq_upload_url` is
|
||||
configured) specifically so this module can be compiled/enabled for its
|
||||
USB-host/bring-up behavior without requiring a backend that doesn't
|
||||
exist.
|
||||
|
||||
A real product decision here — genuinely open, not resolved by this
|
||||
workstream — is whether a POST-per-chunk model is even right versus a
|
||||
persistent WebSocket stream (the spec's `/ws/device-feed` design already
|
||||
establishes a WS pub/sub pattern on the backend for telemetry; a
|
||||
`/ws/device-iq` sibling might fit that architecture better than repeated
|
||||
HTTP POSTs, especially if backpressure/flow-control matters, which for a
|
||||
continuous stream it very much does). That's flagged here rather than
|
||||
silently decided, because it depends on backend design judgment as much
|
||||
as firmware judgment.
|
||||
|
||||
### What's unverified — an honest, specific list for real hardware bring-up
|
||||
|
||||
Nothing below has run against real hardware. In rough order of "most
|
||||
likely to break first":
|
||||
|
||||
1. **USB Host Library API surface.** Function names, struct field names,
|
||||
and callback signatures (`usb_host_client_config_t`'s `.async.
|
||||
client_event_callback` shape in particular — ESP-IDF has changed this
|
||||
API's shape across versions) are written from documented/remembered
|
||||
API shape, not checked against a real ESP-IDF checkout (none available
|
||||
in this environment). **First thing a real bring-up needs: does this
|
||||
even compile against the ESP-IDF version Workstream I's project
|
||||
targets?**
|
||||
2. **RTL2832U enumeration over a real ESP32-P4 USB-OTG host port** — does
|
||||
`usb_host_device_open()` / `usb_host_get_device_descriptor()` actually
|
||||
see the dongle at all on this specific chip's USB-OTG controller (power
|
||||
delivery to the dongle over USB-host mode is itself a hardware
|
||||
question — does the ESP32-P4 dev board supply VBUS in host mode, does
|
||||
the dongle draw more current than it can supply).
|
||||
3. **The init vendor-command sequence itself** — carried over unchanged
|
||||
from `sdr.ts`, which is *itself* unverified. Two layers of "reasoned,
|
||||
not tested." A logic analyzer / USB protocol analyzer trace against a
|
||||
real dongle (or cross-checking against real librtlsdr `-T` verbose
|
||||
output) is needed to confirm register values, not just transfer
|
||||
plumbing.
|
||||
4. **R820T PLL frequency math and demod resample-ratio math** — both are
|
||||
the same simplified integer-N approximation `sdr.ts` uses (its own
|
||||
comment: "real librtlsdr computes the exact sdm/vco from a 28.8MHz
|
||||
crystal reference. Simplified... Marked for hardware."). Likely wrong
|
||||
or imprecise until replaced with the real librtlsdr formula and
|
||||
checked against an actual received signal.
|
||||
5. **Bulk transfer chunk size, pipeline depth, and timeout=0 choice**
|
||||
(`RTLSDR_EXP_BULK_CHUNK_BYTES`/`RTLSDR_EXP_BULK_QUEUE_DEPTH` Kconfig,
|
||||
defaults 16384B / depth 4) — these are guesses. Real ESP32-P4 USB
|
||||
Host Library heap/DMA limits, achievable sustained throughput at
|
||||
2.048 Msps (~4.1 MB/s), and whether `timeout_ms = 0` (no timeout, i.e.
|
||||
"block until data or disconnect") is even the right transfer mode for
|
||||
this endpoint all need real measurement.
|
||||
6. **Drop-on-full backpressure policy** (`iq_blocks_dropped` stat) — is
|
||||
silently dropping IQ blocks when the forward task falls behind
|
||||
acceptable, or does it need real flow control (e.g. throttling the
|
||||
bulk read rate itself)? Depends on (5) and on real WiFi uplink
|
||||
bandwidth from a real device on a real seeker's home network.
|
||||
7. **The proposed IQ-forwarding wire format and endpoint** — entirely
|
||||
hypothetical (see architecture section above); no backend exists to
|
||||
validate the framing against, and the POST-vs-WebSocket question above
|
||||
is unresolved.
|
||||
8. **Concurrent USB Host Library ownership with the rest of the firmware
|
||||
project.** `rtlsdr_exp_start()`/`rtlsdr_exp_stop()` call `usb_host_
|
||||
install()`/`usb_host_uninstall()` directly, assuming this module is the
|
||||
*only* USB Host client in the project. If Workstream I's skeleton (or
|
||||
anything else) also needs USB host for something, this needs to change
|
||||
to a shared-ownership model (install once, both modules register as
|
||||
clients) — impossible to resolve without seeing that code, flagged
|
||||
here for whoever does the merge.
|
||||
9. **Memory footprint.** Multiple in-flight 16KB bulk transfer buffers
|
||||
plus a forward queue plus `esp_http_client` buffers, alongside whatever
|
||||
Workstream I's WiFi/HTTP/BME280/presence stack already needs, on a
|
||||
single ESP32-P4's RAM — not sized or measured against a real linker
|
||||
map.
|
||||
|
||||
### Build / integration notes
|
||||
|
||||
This component is **off by default** (`RTLSDR_EXP_ENABLE` Kconfig option,
|
||||
default `n`) so it cannot affect Workstream I's core build unless
|
||||
explicitly turned on via `idf.py menuconfig` → "RTL-SDR Experimental
|
||||
Module." To include it in a real project once Workstream I's skeleton
|
||||
exists:
|
||||
|
||||
1. Add this directory's `components/` to the consuming project's
|
||||
`EXTRA_COMPONENT_DIRS` in the top-level `CMakeLists.txt` (or copy
|
||||
`components/rtlsdr_experimental/` into that project's own
|
||||
`components/`).
|
||||
2. From wherever `app_main()` sets up its other sensor drivers, e.g.:
|
||||
```c
|
||||
#include "rtlsdr_experimental.h"
|
||||
|
||||
rtlsdr_exp_config_t sdr_cfg;
|
||||
rtlsdr_exp_config_default(&sdr_cfg);
|
||||
sdr_cfg.iq_upload_url = NULL; /* leave unset until a real backend
|
||||
endpoint exists — see README */
|
||||
sdr_cfg.device_bearer_token = DEVICE_BEARER_TOKEN; /* from
|
||||
main/device_config.h, same token
|
||||
used for the telemetry POST loop */
|
||||
rtlsdr_exp_handle_t sdr_handle;
|
||||
if (rtlsdr_exp_init(&sdr_cfg, &sdr_handle) == ESP_OK) {
|
||||
rtlsdr_exp_start(sdr_handle); /* non-blocking; watches for a dongle */
|
||||
}
|
||||
```
|
||||
3. `idf.py build` — **not run in this environment** (no ESP-IDF
|
||||
toolchain available); this module has only been reasoned about, not
|
||||
compiled. Treat "compiles" as an open question for the next person
|
||||
with a real toolchain, not a claim made here.
|
||||
|
||||
Verified in this environment: none of it, against hardware or a real
|
||||
compiler. What *has* been done: careful structural translation of a
|
||||
real, already-partially-researched protocol (`sdr.ts`) to the documented
|
||||
shape of a real, chip-specific API (ESP32-P4's USB Host Library), with
|
||||
every simplification and open question called out rather than hidden.
|
||||
Reference in New Issue
Block a user