mcp-gateway-nexus/docs/modules/07-secrets-manager.md

# Secrets Manager

> Module 07 · Plane: Cross-cutting · Roadmap phase: 3
> Part of [MCP Nexus architecture](../ARCHITECTURE.md).

## Purpose
The Secrets Manager is Nexus's encrypted vault for the credentials that MCP servers need to talk to real services: API keys, passwords, SSH keys, tokens, and certificates. It stores every `Secret` (Architecture §7) **encrypted at rest** via envelope encryption, resolves **secret references** carried by [Recipes](15-smart-recipes.md)/config templates at install time, and hands the plaintext to the [Auto Installer](03-auto-installer.md) for **runtime injection** into the target MCP container (env var or file mount). It embodies tenet #5 and Architecture §9: secrets are **never returned to agents, never written into config sent to agents, and never logged** — unless an explicit policy permits it.

## Responsibilities
- Store `Secret` objects encrypted at rest in the **Secrets** store (Architecture §8), keyed by a stable name/ref.
- Perform **envelope encryption**: a per-secret data-encryption key (DEK) encrypts the payload; the DEK is wrapped by a master **key-encryption key (KEK)**. The KEK provider is pluggable (tenet #3): local master key (from file/env/OS keyring) first, with cloud KMS / HashiCorp Vault adapters later.
- Resolve **secret refs** (e.g. `secret://homeassistant/token`) to plaintext **only** at install/inject time, for the Installer, in memory.
- Provide **runtime injection** material to the Installer: env pairs or tmpfs-backed file mounts, so the plaintext lands inside the sandboxed container and nowhere else.
- **Rotate** secrets and re-wrap DEKs on KEK rotation without re-encrypting every payload; version secrets so an in-flight instance keeps working during rotation.
- **Audit every access** — who/what resolved which secret ref, when, and for which instance — to the append-only log (Architecture §9).
- Enforce a **redaction/leak policy**: refuse to serialize plaintext into any agent-visible payload, dashboard response, or log line; return refs or masked values instead.

## Non-goals
- **Authenticating callers to Nexus.** That is [Authentication](06-authentication.md); this module secures the credentials Nexus *uses*, not access *to* Nexus.
- **Deciding which tools a principal may use.** That is [RBAC](08-rbac.md).
- **Creating containers or templating config.** The [Auto Installer](03-auto-installer.md) injects; [Recipes](15-smart-recipes.md) declare refs. This module only stores, encrypts, and resolves.
- **Being a general end-user password manager or a public KMS.** It is scoped to MCP-server credentials.
- **Discovering credentials.** Discovery methods reference secrets by ref (Module 01); they do not populate the vault automatically.

## Interfaces
```go
// A stored secret. Payload is always encrypted at rest; plaintext exists only in memory.
type Secret struct {
    Ref       string    // "secret://homeassistant/token" — the reference key
    Type      SecretType // apikey | password | ssh-key | token | certificate
    Version   int        // incremented on rotation
    CreatedAt time.Time
    Policy    AccessPolicy // who/what may resolve; may plaintext ever be revealed?
}

type SecretType string

// The store never returns Secret.plaintext except via Resolve, and never logs it.
type Store interface {
    Put(ctx context.Context, ref string, plaintext []byte, t SecretType, p AccessPolicy) error
    // Resolve returns plaintext IN MEMORY for injection; every call is audited.
    Resolve(ctx context.Context, ref string, requester Principal) ([]byte, error)
    Rotate(ctx context.Context, ref string, newPlaintext []byte) (version int, err error)
    Delete(ctx context.Context, ref string) error
    List(ctx context.Context) ([]SecretMeta, error) // metadata only — never plaintext
}

// Envelope encryption: a KEK provider wraps/unwraps per-secret DEKs (tenet #3).
type KEKProvider interface {
    Name() string                                   // "local", "aws-kms", "vault", ...
    WrapDEK(ctx context.Context, dek []byte) (wrapped []byte, keyID string, err error)
    UnwrapDEK(ctx context.Context, wrapped []byte, keyID string) (dek []byte, err error)
}

// What the Installer receives — injection material, not persisted plaintext.
type Injection struct {
    Env    map[string]string // VAR -> value (in-memory, container env)
    Files  map[string][]byte // mountPath -> content (tmpfs file mount)
}

type Injector interface {
    // Resolve all refs in a rendered config into injection material for one instance.
    Materialize(ctx context.Context, refs []string, instanceID string) (Injection, error)
}
```

HTTP/API surface (control-plane API, behind Auth, RBAC-gated; Admin/operator only):
- `POST /api/v1/secrets` — create a secret (write-only; plaintext in, never echoed back).
- `GET  /api/v1/secrets` · `GET /api/v1/secrets/{ref}` — **metadata only** (type, version, created, last-accessed); never plaintext.
- `POST /api/v1/secrets/{ref}/rotate` — rotate; returns new version, not the value.
- `DELETE /api/v1/secrets/{ref}` — delete (guarded against deleting in-use refs).
- `POST /api/v1/secrets/rekey` — rotate the KEK and re-wrap all DEKs.

There is **no** endpoint, MCP tool, or dashboard view that returns a stored plaintext to a caller by default.

## Data
- **Reads/writes** the **Secrets** store (Architecture §8): `{ref, type, version, ciphertext, wrapped_dek, kek_key_id, iv, policy, created_at, last_accessed}` plus envelope-key metadata. Only ciphertext + wrapped DEKs are persisted.
- The **master KEK** is *not* stored in the database; it comes from the configured `KEKProvider` (local key material outside the DB, or an external KMS handle).
- **Reads** secret refs declared by [Recipes](15-smart-recipes.md)/config templates ([Registry](02-package-registry.md)) at install time.
- **Writes** every access to the **Audit** store (append-only) — ref, requester, instance, outcome; never the value.

Secret-ref resolution + injection flow (install time, control plane):

```
1. Reconciler decides to install an MCPInstance; Recipe/config template
     carries refs, e.g. env HASS_TOKEN = secret://homeassistant/token.
2. Installer calls Injector.Materialize(refs, instanceID).
3. For each ref: Store.Resolve(ref, requester)
     a. load {ciphertext, wrapped_dek, kek_key_id} from the Secrets store
     b. KEKProvider.UnwrapDEK(wrapped_dek, kek_key_id) → DEK (in memory)
     c. decrypt ciphertext with DEK → plaintext (in memory only)
     d. audit {ref, requester, instanceID, outcome} — never the value
4. Materialize returns Injection{Env, Files} (plaintext held transiently).
5. Installer creates the sandboxed container with env pairs / tmpfs mounts.
6. Plaintext buffers cleared; it now lives only inside the container.
```

## Dependencies
- Injected by the [Auto Installer](03-auto-installer.md) into sandboxed containers ([Runtime](19-security.md)) at create time.
- Referenced by [Recipes](15-smart-recipes.md) and [Package Registry](02-package-registry.md) config templates via secret refs.
- Access requests carry a `Principal` from [Authentication](06-authentication.md); resolution policy may consult [RBAC](08-rbac.md).
- KEK providers are plugin points via the [Plugin System](11-plugin-system.md) (KMS/Vault adapters, Phase 5).
- Rotation events surface through [Notifications](13-notifications.md); leak-prevention is part of [Security](19-security.md) defense-in-depth.

## Failure modes & handling
| Failure | Behavior |
|---|---|
| KEK provider (KMS) unavailable | Cannot unwrap DEKs → new installs needing secrets **fail closed** and are retried; already-running instances keep their injected material (data plane unaffected). |
| Secret ref not found at install | Installer receives a typed `ErrSecretNotFound`; the instance is not created with a blank/placeholder credential; reconciler flags it for review. |
| Attempt to log/serialize plaintext | Blocked by a redaction wrapper; the value is masked (`****`) and a leak-attempt warning is audited. |
| Rotation mid-use | Versioned secrets: the running instance keeps its injected version; new/restarted instances get the new version; old version retired after drain. |
| KEK compromise suspected | `rekey` re-wraps all DEKs under a new KEK; payloads need not be re-encrypted; event audited + alerted. |
| Corrupt ciphertext / failed decrypt | Fail closed; surface as an integrity error; never return partial/garbage plaintext. |
| Delete of an in-use ref | Rejected unless forced; forcing warns that bound instances will fail on next restart. |

## Security notes
Honors Architecture §9 and tenet #5 as its core contract. **Encryption at rest** is envelope-based: unique per-secret DEK, DEK wrapped by a pluggable KEK/KMS; the master key never lives in the database. Plaintext exists **only transiently in memory** during `Resolve`/`Materialize` and inside the target sandboxed container (preferably tmpfs file mounts over env vars, since env is visible to child processes and some inspection paths). Secrets are **never** returned to agents, never placed in config sent to agents, and never logged — a redaction layer enforces this at the serialization boundary, and any attempt is audited. Every resolution is audited with requester + target instance. Access is RBAC/policy-gated; reading a stored plaintext back out is not a supported operation by default (write-and-inject, not read-back).

## Open questions
- Default injection mechanism: tmpfs file mounts vs env vars — trade off broad MCP-server compatibility against env-visibility risk.
- Policy grammar for the rare "agent may read this secret" exception — how is it expressed and how loudly is it audited?
- KEK rotation cadence and whether re-keying should be automatic on a schedule.
- Do we support external secret *sources* (read-through to Vault/KMS at inject time) in addition to storing our own ciphertext?
- Certificate lifecycle: does the Secrets Manager track expiry and drive renewal, or is that the [Update Manager](10-update-manager.md)?

## Milestone
Delivered in **Phase 3** (Secure). Thin slice first: local-KEK envelope encryption, `Put`/`Resolve`/`Rotate`, secret-ref resolution, and **runtime injection into a sandboxed container** by the Installer, with full audit and hard redaction on logs/agent payloads. KMS/Vault KEK providers are Phase 5 plugins. Contributes directly to the Phase 3 exit proof: a secret-backed MCP works without the secret ever appearing in any agent-visible payload or log.