mcp-gateway-nexus/docs/modules/07-secrets-manager.md

Secrets Manager

Module 07 · Plane: Cross-cutting · Roadmap phase: 3 Part of MCP Nexus architecture.

Purpose

The Secrets Manager is Nexus's encrypted vault for the credentials that MCP servers need to talk to real services: API keys, passwords, SSH keys, tokens, and certificates. It stores every Secret (Architecture §7) encrypted at rest via envelope encryption, resolves secret references carried by Recipes/config templates at install time, and hands the plaintext to the Auto Installer for runtime injection into the target MCP container (env var or file mount). It embodies tenet #5 and Architecture §9: secrets are never returned to agents, never written into config sent to agents, and never logged — unless an explicit policy permits it.

Responsibilities

• Store Secret objects encrypted at rest in the Secrets store (Architecture §8), keyed by a stable name/ref.
• Perform envelope encryption: a per-secret data-encryption key (DEK) encrypts the payload; the DEK is wrapped by a master key-encryption key (KEK). The KEK provider is pluggable (tenet #3): local master key (from file/env/OS keyring) first, with cloud KMS / HashiCorp Vault adapters later.
• Resolve secret refs (e.g. secret://homeassistant/token) to plaintext only at install/inject time, for the Installer, in memory.
• Provide runtime injection material to the Installer: env pairs or tmpfs-backed file mounts, so the plaintext lands inside the sandboxed container and nowhere else.
• Rotate secrets and re-wrap DEKs on KEK rotation without re-encrypting every payload; version secrets so an in-flight instance keeps working during rotation.
• Audit every access — who/what resolved which secret ref, when, and for which instance — to the append-only log (Architecture §9).
• Enforce a redaction/leak policy: refuse to serialize plaintext into any agent-visible payload, dashboard response, or log line; return refs or masked values instead.

Non-goals

• Authenticating callers to Nexus. That is Authentication; this module secures the credentials Nexus uses, not access to Nexus.
• Deciding which tools a principal may use. That is RBAC.
• Creating containers or templating config. The Auto Installer injects; Recipes declare refs. This module only stores, encrypts, and resolves.
• Being a general end-user password manager or a public KMS. It is scoped to MCP-server credentials.
• Discovering credentials. Discovery methods reference secrets by ref (Module 01); they do not populate the vault automatically.

Interfaces

// A stored secret. Payload is always encrypted at rest; plaintext exists only in memory.
type Secret struct {
    Ref       string    // "secret://homeassistant/token" — the reference key
    Type      SecretType // apikey | password | ssh-key | token | certificate
    Version   int        // incremented on rotation
    CreatedAt time.Time
    Policy    AccessPolicy // who/what may resolve; may plaintext ever be revealed?
}

type SecretType string

// The store never returns Secret.plaintext except via Resolve, and never logs it.
type Store interface {
    Put(ctx context.Context, ref string, plaintext []byte, t SecretType, p AccessPolicy) error
    // Resolve returns plaintext IN MEMORY for injection; every call is audited.
    Resolve(ctx context.Context, ref string, requester Principal) ([]byte, error)
    Rotate(ctx context.Context, ref string, newPlaintext []byte) (version int, err error)
    Delete(ctx context.Context, ref string) error
    List(ctx context.Context) ([]SecretMeta, error) // metadata only — never plaintext
}

// Envelope encryption: a KEK provider wraps/unwraps per-secret DEKs (tenet #3).
type KEKProvider interface {
    Name() string                                   // "local", "aws-kms", "vault", ...
    WrapDEK(ctx context.Context, dek []byte) (wrapped []byte, keyID string, err error)
    UnwrapDEK(ctx context.Context, wrapped []byte, keyID string) (dek []byte, err error)
}

// What the Installer receives — injection material, not persisted plaintext.
type Injection struct {
    Env    map[string]string // VAR -> value (in-memory, container env)
    Files  map[string][]byte // mountPath -> content (tmpfs file mount)
}

type Injector interface {
    // Resolve all refs in a rendered config into injection material for one instance.
    Materialize(ctx context.Context, refs []string, instanceID string) (Injection, error)
}

HTTP/API surface (control-plane API, behind Auth, RBAC-gated; Admin/operator only):

• POST /api/v1/secrets — create a secret (write-only; plaintext in, never echoed back).
• GET /api/v1/secrets · GET /api/v1/secrets/{ref} — metadata only (type, version, created, last-accessed); never plaintext.
• POST /api/v1/secrets/{ref}/rotate — rotate; returns new version, not the value.
• DELETE /api/v1/secrets/{ref} — delete (guarded against deleting in-use refs).
• POST /api/v1/secrets/rekey — rotate the KEK and re-wrap all DEKs.

There is no endpoint, MCP tool, or dashboard view that returns a stored plaintext to a caller by default.

Data

• Reads/writes the Secrets store (Architecture §8): {ref, type, version, ciphertext, wrapped_dek, kek_key_id, iv, policy, created_at, last_accessed} plus envelope-key metadata. Only ciphertext + wrapped DEKs are persisted.
• The master KEK is not stored in the database; it comes from the configured KEKProvider (local key material outside the DB, or an external KMS handle).
• Reads secret refs declared by Recipes/config templates (Registry) at install time.
• Writes every access to the Audit store (append-only) — ref, requester, instance, outcome; never the value.

Secret-ref resolution + injection flow (install time, control plane):

1. Reconciler decides to install an MCPInstance; Recipe/config template
     carries refs, e.g. env HASS_TOKEN = secret://homeassistant/token.
2. Installer calls Injector.Materialize(refs, instanceID).
3. For each ref: Store.Resolve(ref, requester)
     a. load {ciphertext, wrapped_dek, kek_key_id} from the Secrets store
     b. KEKProvider.UnwrapDEK(wrapped_dek, kek_key_id) → DEK (in memory)
     c. decrypt ciphertext with DEK → plaintext (in memory only)
     d. audit {ref, requester, instanceID, outcome} — never the value
4. Materialize returns Injection{Env, Files} (plaintext held transiently).
5. Installer creates the sandboxed container with env pairs / tmpfs mounts.
6. Plaintext buffers cleared; it now lives only inside the container.

Dependencies

• Injected by the Auto Installer into sandboxed containers (Runtime) at create time.
• Referenced by Recipes and Package Registry config templates via secret refs.
• Access requests carry a Principal from Authentication; resolution policy may consult RBAC.
• KEK providers are plugin points via the Plugin System (KMS/Vault adapters, Phase 5).
• Rotation events surface through Notifications; leak-prevention is part of Security defense-in-depth.

Failure modes & handling

Failure	Behavior
KEK provider (KMS) unavailable	Cannot unwrap DEKs → new installs needing secrets fail closed and are retried; already-running instances keep their injected material (data plane unaffected).
Secret ref not found at install	Installer receives a typed ErrSecretNotFound; the instance is not created with a blank/placeholder credential; reconciler flags it for review.
Attempt to log/serialize plaintext	Blocked by a redaction wrapper; the value is masked (****) and a leak-attempt warning is audited.
Rotation mid-use	Versioned secrets: the running instance keeps its injected version; new/restarted instances get the new version; old version retired after drain.
KEK compromise suspected	rekey re-wraps all DEKs under a new KEK; payloads need not be re-encrypted; event audited + alerted.
Corrupt ciphertext / failed decrypt	Fail closed; surface as an integrity error; never return partial/garbage plaintext.
Delete of an in-use ref	Rejected unless forced; forcing warns that bound instances will fail on next restart.

Security notes

Honors Architecture §9 and tenet #5 as its core contract. Encryption at rest is envelope-based: unique per-secret DEK, DEK wrapped by a pluggable KEK/KMS; the master key never lives in the database. Plaintext exists only transiently in memory during Resolve/Materialize and inside the target sandboxed container (preferably tmpfs file mounts over env vars, since env is visible to child processes and some inspection paths). Secrets are never returned to agents, never placed in config sent to agents, and never logged — a redaction layer enforces this at the serialization boundary, and any attempt is audited. Every resolution is audited with requester + target instance. Access is RBAC/policy-gated; reading a stored plaintext back out is not a supported operation by default (write-and-inject, not read-back).

Open questions

• Default injection mechanism: tmpfs file mounts vs env vars — trade off broad MCP-server compatibility against env-visibility risk.
• Policy grammar for the rare "agent may read this secret" exception — how is it expressed and how loudly is it audited?
• KEK rotation cadence and whether re-keying should be automatic on a schedule.
• Do we support external secret sources (read-through to Vault/KMS at inject time) in addition to storing our own ciphertext?
• Certificate lifecycle: does the Secrets Manager track expiry and drive renewal, or is that the Update Manager?

Milestone

Delivered in Phase 3 (Secure). Thin slice first: local-KEK envelope encryption, Put/Resolve/Rotate, secret-ref resolution, and runtime injection into a sandboxed container by the Installer, with full audit and hard redaction on logs/agent payloads. KMS/Vault KEK providers are Phase 5 plugins. Contributes directly to the Phase 3 exit proof: a secret-backed MCP works without the secret ever appearing in any agent-visible payload or log.