# Did the AI leak my fake password? Secrets in AI prompts

URL: https://systemdesignsimple.com/did-the-ai-leak-my-fake-password/
Published: 2026-11-07
Category: AI Systems
Publisher: System Design Simple (https://systemdesignsimple.com)

## Short answer

A secret in an outgoing AI request reaches its recipient even if the reply omits it; logs keep it where body capture is on. In our local simulation a fake password was in 40 of 40 prompts and 40 of 40 body-capture log lines, but 22 of 40 scripted answers. A pre-send redactor with 4 patterns and stable placeholders made it 0 of 40 everywhere; required facts stayed in 40 of 40.

## Key facts

- **Root cause:** A secret in an outgoing request reaches its recipient even if the reply omits it. Logs contain it only where body capture is enabled, so the reply is the least reliable place to look
- **No redaction (lab):** Fake password CANARY-7Q2X-FAKE in 40 of 40 simulated outgoing prompts and 40 of 40 lines of a simulated log that captures bodies; the phone-shaped canary also 40 of 40 in both
- **Scripted answers (lab):** Password in 22 of 40 (per request 1, 9, 8, 4 of 10), phone in 13 of 40. These come from copy chances we chose, so they say nothing about a real model
- **With redaction (lab):** 4 regex patterns, stable placeholders: 0 of 40 in prompts, log lines and answers for both fake values
- **Usefulness (lab):** Required facts present in 40 of 40 redacted scripted answers; the same check fell to 0 of 40 when README.md and notes.md were left out, so it can fail
- **Limit of patterns:** A DB_PASS= line, a JSON password field, a bracketed phone and a bare random token all passed the 4 patterns; only an entropy check caught the token

Secrets in AI prompts are exposed by the request, whatever the reply says. If a password is in the request your app sends, the provider receives it even when the answer never repeats it, and any log that captures request bodies keeps a copy. In our local simulation a fake password was in 40 of 40 outgoing prompts and 40 of 40 body-capture log lines but only 22 of 40 scripted answers, and a pre-send redactor took all three to 0 of 40.

## Where do secrets in AI prompts go? {#how-it-works}

In this API example, the request body carries the prompt messages alongside configuration fields. Your app serializes the messages, usually as JSON, and sends them over HTTPS to a model provider. TLS protects the bytes in transit, and the receiver, or anything in between that terminates TLS, reads them in full.

Applications assemble context from several inputs. Besides what the user typed, they can add retrieved documents ([how RAG finds one line in 100 pages, AI 4](/how-ai-finds-one-line-in-100-pages/)), file contents and tool results (tool calling, AI 5), so a secret can enter a prompt without anyone pasting it. A secret remains in subsequent requests while that history is retained and included ([why an agent re-sends its history, AI 7](/why-an-ai-agent-re-sends-its-history/)).

Whether a copy stays behind depends on how logging is configured. The default Nginx <code>combined</code> access log records the request line, status, size, referer and user agent, and no body; a body appears only if someone adds <code>$request_body</code> to a <code>log_format</code> ([NGINX log module](https://nginx.org/en/docs/http/ngx_http_log_module.html)). Applications may capture prompt bodies elsewhere: debug logs that print the outgoing payload, LLM tracing tools, error reporters that attach the failing request.

OWASP files this under LLM02, Sensitive Information Disclosure, and its first listed mitigation is sanitizing data before it reaches the model.

## The lab: 40 prompts, 22 answers {#lab}

<code>labs/ai-9</code> is a local simulation: Node.js 22, no network, a scripted stand-in model and an in-memory provider log that stores each request body and reply, as a gateway with body capture would. Nothing was transmitted. A three-file project holds two fake canaries: the password <code>CANARY-7Q2X-FAKE</code> twice in <code>config.env</code> (as <code>DB_PASSWORD</code> and inside <code>DATABASE_URL</code>) and the phone-shaped <code>+1-555-0142</code> in free-text notes. Four requests ran 10 times each per setup, seed 9009, with the whole folder packed into every prompt.

[Image: Sketched airport check-in counter with a project folder dropping into an open suitcase, the config.env password line glowing red and a counter reading 40/40 prompts | The folder goes into the suitcase whole: the fake password is in 40 of 40 simulated prompts.]

Output of <code>node labs/ai-9/run.mjs</code>, counts shown as password / phone out of 10:

```text
request       prompt   log      answer   facts  | prompt   log      answer   facts
              --- no redaction ---              | --- with redaction ---
summarize     10/10    10/10    1/2      10/10  | 0/0      0/0      0/0      10/10
debug-config  10/10    10/10    9/1      10/10  | 0/0      0/0      0/0      10/10
setup-guide   10/10    10/10    8/4      10/10  | 0/0      0/0      0/0      10/10
what-i-need   10/10    10/10    4/6      10/10  | 0/0      0/0      0/0      10/10
```

The prompt and log columns follow from the design. The file is in every request and the log stores every body, so 40 of 40 involves no model behaviour. The answer column measures copy chances we chose (from 0.1 for summarize to 0.8 for debug-config), so 22 of 40 says nothing about how often a real model repeats a secret. It does show what a reply-only check sees: 22 where the provider and the log saw 40, which leaves 18 requests that carried the password behind a clean-looking answer.

[Image: A grid of 40 answer slots in four rows of ten, one row per request, with 22 slots red and a counter reading 22/40 answers under a lit counter reading 40/40 prompts | Scripted answers, one row per request: 22 of 40 carried the password, under odds we picked.]

Each request also has required facts read from the true project: versions, the start command, the fix "DB_PORT to 5432" and the key name <code>DB_PASSWORD</code>. Redacted answers kept them in 40 of 40. The check can fail, too: with <code>README.md</code> and <code>notes.md</code> left out it dropped to 0 of 40.

## Redact decoded fields, then serialize {#mechanism}

The redactor runs before anything sends or logs, and it has to see text the way the model will. It parses the request, validates it against an explicit schema, redacts the decoded string in every field it knows (system prompt, message text, tool-call arguments, tool results), rejects any field or part type it does not know, and only then serializes. Rebuild the request from allowed fields. Copying the input with <code>{ ...body }</code> and overwriting the scanned fields forwards whatever the walker skipped, such as a top-level <code>system</code> string or <code>tool_calls[].function.arguments</code>.

Regexes run over the serialized JSON fail twice in the demo. JSON writes a newline as the two characters <code>\n</code>, so the next key follows the word character <code>n</code> and the <code>\b</code> anchor before <code>DB_PASSWORD</code> never matches. And <code>\S+</code> after <code>SERVICE_TOKEN=</code> runs past the closing quote of the <code>system</code> string, since compact JSON has no space there, and swallows the start of <code>messages</code>. The canary survives and the result is no longer valid JSON.

Each value gets a session-stable placeholder. Both copies of the password become the same token, so the model can still tell they are one value and say "change it in both places". Random tokens per occurrence lose that link, and deleting the value invites "your password is empty". Replacement is by position, so <code>TOKEN=TOKEN</code> cannot end up with the key swapped and the credential intact.

```js
// labs/ai-9/redact.mjs (excerpt). Group 1 is the context, group 2 the value.
const PATTERNS = [
  { cls: 'SECRET', re: /(\b[A-Z0-9_]*(?:PASSWORD|SECRET|TOKEN|API_KEY)[A-Z0-9_]*=)(\S+)/g }, // KEY=value
  { cls: 'SECRET', re: /(\/\/[^:/\s]+:)([^@\s]+)@/g },                                     // user:pass@host
  { cls: 'EMAIL',  re: /()([A-Za-z0-9._%+-]+@[A-Za-z0-9-]+\.[A-Za-z.]{2,})/g },
  { cls: 'PHONE',  re: /()(\+?\d{1,3}[-. ]\d{3}[-. ]\d{4})\b/g },
];
const tokenFor = (cls, value) => {
  let e = byValue.get(byToken.get(value) ?? value);        // an issued token resolves to its entry
  if (!e) {
    counts[cls] = (counts[cls] ?? 0) + 1;
    e = { token: `<${cls}_${tag}_${counts[cls]}>`, cls };  // tag: 16 random bytes per session
    byValue.set(value, e);
    byToken.set(e.token, value);
  } else if (RANK[cls] > RANK[e.cls]) {
    e.cls = cls;                                           // promote; the spelling stays
  }
  return e.token;
};
const redactText = (text) => {
  let out = text;
  for (const { cls, re } of PATTERNS) {
    out = out.replace(re, (whole, pre, value) =>
      pre + tokenFor(cls, value) + whole.slice(pre.length + value.length)); // by position
  }
  return out;
};
```

Run on a debug request whose decoded first message holds the config (the random tag printed as <code>TAG</code>). Placeholders the session issued are removed before the entropy scan, because their random tag would trip it too:

```text
Help me debug the config.

DB_PASSWORD=<SECRET_TAG_2>
DATABASE_URL=postgres://shop_app:<SECRET_TAG_2>@db.internal:5433/shop
TOKEN=<SECRET_TAG_3>
- If the database is down, call the on-call phone <PHONE_TAG_1>.
DB_PASS=hunter-fake-22
{"password": "json-fake-pw"}
backup line: (415) 555-0199
payments key in prose: q8Zr2LwT9xVb4NcK7mPd3HsY6fJ1

canary in outgoing body: false | placeholders in local map: 6
regex over serialized JSON: canary left = true | valid JSON = false
entropy detector flags: [ 'q8Zr2LwT9xVb4NcK7mPd3HsY6fJ1 (4.81 bits/char)' ]
```

[Image: A scanner arch labelled checkpoint across the belt before the log camera, stamping the red password card into a grey tag reading SECRET_1 and the phone card into PHONE_1 | The checkpoint swaps both fake values for placeholders before the log and the provider.]

## What four patterns miss {#limits}

The last four lines of that output are formats the patterns were never written for, and all four reached the outgoing request: a key without <code>PASSWORD</code> in its name, a JSON field, a bracketed phone and a bare random token. Production redaction stacks detectors that fail differently:

| Detector | Catches | Blind spot |
| --- | --- | --- |
| Exact token | Your own known secrets and planted canaries | Unknown values; base64, percent-encoded or split copies unless you decode first |
| Shape (regex) | Labelled values, URL userinfo, fixed prefixes such as `ghp_` and `AKIA` | Free-form passwords, unfamiliar layouts |
| Entropy | Random keys no rule names: 28 distinct characters give log2 28 = 4.81 bits per character | Hex entropy is at most log2 16 = 4.0 bits per character, so this demo's strict `> 4.0` test cannot flag hex-only tokens; short tokens |
| Validators, NER | Checksums, phone rules, names in prose | Added latency; models miss and over-match |

Entropy also flags harmless random strings, so a hit alone does not prove a value is a credential.

Gitleaks pairs regex rules with keywords and per-rule entropy, TruffleHog can check whether a found key is live, and Presidio adds context words and a named-entity model for personal data in prose. Keep a corpus of fake secrets in many formats, run the redactor over it in CI, and fail the build when recall drops.

Replies need the same scan before they are logged or displayed. With streamed replies (AI 2), buffer the complete response for these detectors; <code>NOTES.md</code> explains why a fixed holdback is not enough.

## Restoring placeholders safely {#restore}

Keep the value map local to the session. Restore personal data only for its owner into an approved display destination. Secret restoration needs a separate authorized operation; expanding a placeholder inside a tool call can send the value to an attacker through prompt injection. The example uses a single-use write grant and assumes a directory whose parents only you can modify. Random placeholder tags prevent accidental collisions and provide no authorization.

The example keeps the map in process memory. Production code must add session expiry, reject operations after expiry and clear the maps when the session ends. The full restore rules are in <code>labs/ai-9/NOTES.md</code>.

## Our take {#verdict}

Scope comes before redaction: keep <code>.env</code> files, keys and credential stores out of the context and out of an agent's readable paths (agent file permissions, AI 8). Put the redactor at the last point before the network, at the client for secrets it knows exactly and at an egress gateway as a floor for every app, and make the gateway's own logs record metadata only. Four regexes are enough for a demo; with real data, watch recall on your own formats.

| Option | Good for | Costs you |
| --- | --- | --- |
| No redaction, check replies only | Nothing | Sent when included; logged where body capture is enabled. Lab: 40 of 40 prompts while replies showed 22 |
| Scope the context | Removes the secret at the source | You must know which files a task needs; the facts check fell to 0 of 40 without them |
| Client-side redaction | Plaintext stays on the device; exact match on the user's own secrets | Every client carries and updates detectors |
| Egress gateway redaction | One policy for every app, enforced by network rules | The gateway sees plaintext and holds the restore map; it must redact before its own logging |

If a real credential reached an unintended recipient, revoke it promptly, issue and deploy a replacement, and review its usage during the exposure window ([OWASP remediation](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html#92-remediation)). Purging your own logs cannot recall a copy that already left. Gateway sizing and the answer-count statistics are in <code>labs/ai-9/NOTES.md</code>.

Every value in labs/ai-9 is fake. The numbers show where a secret goes once it is in a prompt, not the behaviour of any company's model, product or private systems.

**Read next:** how a cloud coding session can use an API key it never sees, with network secrets in Claude Code cloud environments.

## Frequently asked questions

**If the AI did not repeat my password, is it safe?**
No. If the password was in the request that was sent, the recipient has it whatever the reply says, and any log with body capture enabled keeps a copy.

**Does redaction catch every secret?**
No. Detectors find only the formats they were written for: our 4 patterns missed a DB_PASS= line, a JSON password field and a bracketed phone, so layer detectors and rotate anything that was sent.

**Did the experiment use a real AI model?**
No. labs/ai-9 is a local simulation with a scripted stand-in model and a simulated log; nothing was transmitted, and the answer counts come from copy chances we chose.

## Sources

- [OWASP Top 10 for LLM Applications 2025: LLM02 Sensitive Information Disclosure](https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/)
- [OWASP Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html)
- [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html)
- [NGINX ngx_http_log_module](https://nginx.org/en/docs/http/ngx_http_log_module.html)
- [Gitleaks (GitHub repository and rule format)](https://github.com/gitleaks/gitleaks)
- [TruffleHog (GitHub repository: detectors and verification)](https://github.com/trufflesecurity/trufflehog)
- [Microsoft Presidio: PII detection and anonymization](https://microsoft.github.io/presidio/)