API Key Security for AI Tooling

Trust boundary

You can hide the raw key from an agent only by moving it behind a different trusted boundary.

If the agent can run arbitrary shell commands, Python, or network code in the same environment that holds your provider key, then the key is effectively available to that agent. Environment variables, dotfiles, and "encrypted then decrypted in the same process" do not change that.

The safer pattern is a split design: a separate trusted launcher, broker, or backend holds the key and exposes only a constrained capability interface to the agent.

Choose the model surface on model access. This page starts later: once you already know which integration surface you are protecting.

Choose the boundary

Pick the strongest split you can actually run.

I only have a raw API key Start with the beginner path. Learn the trust-boundary idea first, then use the localhost broker as your next step up from a plain shell variable. I already use 1Password or Key Vault Skip to secret-manager integration. Keep your current secret store. The broker is still useful, but it should read from that store instead of replacing it. I already have backend infrastructure Prefer the backend path. A remote backend or managed identity is usually stronger than a localhost broker for production systems.

The real distinction

Storage secrecy is not the same thing as runtime secrecy.

Environment variable

Good for not hardcoding a secret into source. Not good for hiding a secret from an agent that can inspect the shell environment or run code as your user.

Encrypted file or memory

Good for protecting a secret at rest. Not enough if the same process or same user context must decrypt it and the agent can inspect that runtime.

Separate broker or backend

This is the meaningful isolation pattern. A different trusted service holds the key and the agent gets only a narrow interface.

Security patterns

Only some patterns actually change what the agent can see.

Pattern	Can the agent usually read the raw key?	What it is good for	Main limitation
Key in `.env` or shell environment	Usually yes	Quick local experimentation	If the agent can inspect files or env vars, the secret is exposed.
Key encrypted on disk, then decrypted by the same runtime	Usually yes at runtime	Protecting a laptop or disk at rest	Does not protect against an agent inside the same active runtime boundary.
OS keychain plus trusted launcher	Sometimes no	Human-operated local tools with better secret hygiene	Only helps if the agent cannot call the keychain or inspect the launcher process.
Local broker on `localhost`	Often no	Letting an agent use model access through a restricted interface	The broker itself becomes critical infrastructure and must enforce limits.
Remote backend with managed identity or secret manager	Often no	Production systems, org policy, auditability, rotation	The agent still gets a capability; the backend must constrain what it can do.
Hosted subscription product	Not applicable	Human-facing assistants where you never hold the provider key directly	You are delegating trust to the product surface rather than integrating the raw API yourself.

Practical defaults

Use the simplest pattern that matches your actual trust goal.

Acceptable beginner posture

Put the key in one temporary shell session and use it only to prove one boring request. This is fine for learning, but assume an agent with shell access could read it.

Better local posture

A human-trusted launcher reads the key from an OS keychain and starts a small local broker. The agent receives only the broker endpoint, not the provider key itself.

Runnable example: local credential broker side path.

Best production posture

Put the secret behind your own backend or managed identity layer and let the agent call only your service. Treat raw provider keys as infrastructure secrets, not agent context.

Capability split

The clean mental model is a capability broker.

You
  -> unlock keychain or secret manager
  -> start trusted broker

Agent / CLI host
  -> calls localhost broker or your backend
  -> never receives raw provider key

Broker / backend
  -> holds provider credential
  -> enforces allowlists, quotas, logs, and approval rules
  -> calls the model provider API

This is the important conceptual answer to your question: the relationship does not need to be fully co-conspiratorial. Some trusted component must know the key, but it does not have to be the agent.

Raw-key path

If all you have is a raw key, start by understanding the risk plainly.

What most people try first

They put the key in .env, export it into a shell, or paste it into a tool config file. That is common and understandable. It also means any agent with that same shell or file access may be able to read the secret.

What to do next

Use the raw key only long enough to prove one request, then move to a broker pattern so the tool host talks to localhost instead of carrying the provider credential directly.

Local split

Use a localhost broker when you want a practical local split.

The broker is not "better than a secret manager." It solves a different problem. A secret manager stores the secret well. A broker changes which runtime gets to use it directly.

Also, 127.0.0.1 is not a magic shield. Loopback binding narrows exposure, but another local app, a browser-triggered request, or same-user automation may still reach the broker unless the capability call itself is protected.

python3 labs/13-local-broker/broker.py --config labs/13-local-broker/broker_config.dry-run.json

Optional side path: put model access behind a localhost broker.

What it gives you

A narrow local endpoint, allowed-path controls, minimal logs, and a place to attach the bearer token outside the agent host.

What it does not give you

It does not make an unsafe machine safe, and it does not eliminate the need for scoped credentials, rotation, or local OS security.

When to use it

Use it for local experimentation, educational tooling, or when you want your host to call a constrained local capability instead of owning the provider key.

What to add in a real local setup

Keep the broker on 127.0.0.1, keep the path allowlist narrow, and require a separate caller token such as X-Broker-Token loaded from a trusted launcher.

What this still does not solve

If hostile code can run as the same user and read the same launcher state or local token source, you still need a stronger boundary such as a backend, separate user, or managed identity path.

Secret-store path

If your key already lives in 1Password or Azure Key Vault, keep that layer.

1Password and Azure Key Vault are usually superior to local dotfiles or plain environment variables for secret storage and rotation. The broker should sit after that storage layer, not replace it.

If your secret is in...	Use the broker like this	Why this is stronger
1Password	Have the broker fetch the secret with a non-shell command such as `op read ...` at startup.	The agent still talks only to the broker. The raw key stays in 1Password rather than a checked-in config or shell history.
Azure Key Vault	Have the broker fetch the secret via the Azure CLI or a managed-identity-backed helper command.	You keep org policy, rotation, and access control at the vault layer while changing runtime visibility locally.
OS keychain	Unlock it as the human, then start the broker from a trusted launcher.	The host or agent can use the broker capability without directly carrying the provider credential.

{
  "mode": "proxy",
  "upstream_base_url": "https://api.openai.com",
  "allowed_paths": ["/v1/chat/completions"],
  "client_token_env": "BROKER_CLIENT_TOKEN",
  "secret_source": {
    "kind": "command",
    "argv": ["op", "read", "op://Private/OpenAI/api key"]
  }
}

{
  "mode": "proxy",
  "upstream_base_url": "https://api.openai.com",
  "allowed_paths": ["/v1/chat/completions"],
  "client_token_env": "BROKER_CLIENT_TOKEN",
  "secret_source": {
    "kind": "command",
    "argv": ["az", "keyvault", "secret", "show", "--vault-name", "MY_VAULT", "--name", "openai-api-key", "--query", "value", "-o", "tsv"]
  }
}

In both cases, the host should present a separate broker token to call localhost. Keep that caller token distinct from the provider credential.

Checked-in example configs: broker_config.1password.example.json and broker_config.azure-keyvault.example.json.

Backend split

A backend or managed identity is usually stronger than localhost.

If you already have backend infrastructure, the cleanest move is often a remote service boundary: your app or agent calls your backend, and your backend holds the real provider credential or managed identity. That is usually better for multi-user systems, auditability, and central policy.

The localhost broker remains useful as a teaching tool and as a local developer pattern, but it is not the end state for most production organizations.

Practice companion: backend broker pattern.

Small habits

Small habits that still matter.

Do

Use dedicated keys, rotate them, scope them when the provider allows it, keep them out of git, and avoid logging full auth headers or raw request dumps.

If the agent can edit files or run shell commands, assume it can read the secrets available to that same user context unless you have explicitly split the trust boundary.

Do not rely on

"It is encrypted somewhere," "it is only in memory," or "the agent is polite." Those do not create real isolation if the key is still usable from the same runtime boundary.

Also avoid teaching people to paste secrets directly into prompts, checked-in config files, notebooks, or shared shell history.

Use this with

Pair this page with model access, labs bootstrap, and lab 00.

Optional implementation path: the local broker side path.

Related concepts

Hooks, approvals, and narrow tool interfaces matter here because they are part of how you keep a capability smaller than the raw secret behind it.