What it looks like
Say your application builds this prompt:How it works
docker run, root token DEMO — so every call below is runnable as written.
Step 1: Find the personal data
Regular expressions handle values with a fixed shape. Order them most specific first, and discard any match that overlaps a span already taken, or a loose pattern will claim a span a precise one should own.SSN after PHONE in that list and 123-45-6789 is tokenized as a phone number — the digit pattern matches it too, and whichever runs first wins.
Step 2: Tokenize the values
How a value becomes a placeholder depends on where it came from: discovered in free text, or already held as a user record. Pick the tab that matches your prompt — the two take different calls. The Python snippets throughout use one small HTTP helper:- Found in free text
- The user profile is known
The values come from Each row echoes the Two parameters matter here:
find_pii above, so each label is whatever the matching pattern was named.record it came from, which is what lets you map tokens back onto the spans you found. string is stored under its canonical name text, and only creditcard carries a tokenbase.Which gives the placeholders the prompt is built from:-
unique: truereturns the same token for a value already in the vault, so an email keeps one identity across every turn of a conversation and across restarts. Without it each call mints a new token and the model sees the same person as a different one each time. Send a value again and it comes back underduplicates, carrying the original token: -
finaltimeexpires the token without a cleanup call. A prompt token rarely needs to outlive the conversation — see Retention below.
[LABEL:uuid] in place of every value. The label is what lets the model reason — it can write an email address into a sentence about sending mail without ever seeing one. The UUID is what makes detokenization stateless: the placeholder carries its own identity, so any process holding vault access can resolve it later, with nothing kept on the side in between.
Step 3: Build the prompt and call the model
Substitute back-to-front so earlier replacements do not shift the offsets of later spans.Step 4: Detokenize the reply
The model’s reply comes back carrying the same placeholders. Resolving them is one regex pass plus a source of plaintext — and the source is whichever tab you took in Step 2.- Resolve through the vault
- Resolve from a prefetched profile
Bulk detokenization returns plaintext, so it takes a short-lived unlock UUID as a second authorization step: call An This is the general case: it resolves any token, including ones minted in an earlier turn or by another service.
BulkListUnlock, then pass the unlockuuid to BulkListTokens.unlockuuid is good for one bulk session — fetch a fresh one each time rather than caching it.In Python, collect the distinct UUIDs the model echoed back and resolve them in one call:vault.get(uuid, m.group(0)) fallback. It fails closed: an unresolvable token is never silently replaced with the wrong value.
The complete round trip
The four steps compose into the exchange shown at the top of this page:tokenize finds the four values and swaps in placeholders, ask sends a prompt carrying no personal data, and detokenize restores the values in the answer. Nothing personal is written to your logs, your traces, or the provider’s.
Retention
Give prompt tokens an expiry. A token minted for a conversation rarely needs to outlive it, and expiry means a retention limit does not depend on your bookkeeping being correct:finaltime elapses the token reports token expired and stops resolving, with no cleanup call from you. slidingtime restarts the window on every access instead, which suits a long-running conversation. To remove a token early, use TokenDelete for one or BulkDeleteTokens — which takes the same unlockuuid as BulkListTokens — for many.
Because
unique: true returns an existing token, a value already in the vault keeps the expiry it was created with. The finaltime you send on a repeat call does not extend it.What this does not protect
Tokenization narrows the blast radius. It does not make the problem go away, and a security reviewer will ask about each of these.- Tokenized data is still personal data. This is pseudonymization, not anonymization. Under GDPR the tokenized prompt remains in scope, because you hold the key that reverses it. It reduces exposure to the model provider; it does not take the workload out of scope.
- Detection sets the ceiling. Everything the patterns miss is sent in the clear. Names, employers, and locations need NER, and NER is not perfect either.
- Context can re-identify. “The patient in room 4 admitted Tuesday” identifies someone with no tokenizable field present.
unique: truemakes it worse in one specific way: equal values produce equal tokens, so anyone holding two prompts can tell they concern the same person without resolving anything. That is the price of cross-turn consistency — see why shared records, not long-lived tokens. - The model cannot use what it cannot see. It cannot validate an address, sort by surname, or judge whether an email looks like a throwaway domain. If the task genuinely needs the value, tokenizing that field breaks the task.
- Detokenization is a privilege boundary. Whatever resolves the reply holds a credential that turns tokens back into plaintext. Scope it with an access-control policy rather than using the root token, and keep the detokenizing step out of any component that forwards text onward.
- The audit trail records the call, not the token.
TokenGet,BulkListUnlock, andBulkListTokenseach write an audit event with the caller identity, tenant, timestamp, and status — enough to show who detokenized and when, but not which values they resolved.
Why a vault, and not a dictionary?
This pattern does not obviously need a vault. The direct implementation keeps a map in memory, filled when you build the prompt and read when the reply comes back:
The vault is not free, and the honest entries run the other way too:
So the question is not which is better in the abstract. It is whether the mapping between a placeholder and a real person is something your application can afford to lose, leak, or fail to erase. Once it is state you have to keep — across turns, across services, across an audit — you have started building a vault, and the build-or-buy trade-off applies in full.
Related
- Store user fields — whole-profile records versus per-field tokens
- Format-preserving tokenization — token types, expiry, bulk operations
- Access control — scoping who may detokenize
- Licensing and limits — what counts as a record