How do you keep the same PII token across Snowflake and Databricks?
You apply the identical keyed transform, with one shared key, before or independent of the warehouse. The join-preserving behavior comes entirely from that discipline.
Deterministic tokenization means the same plaintext under the same key always produces the same token. Google Cloud's Sensitive Data Protection docs put it plainly: with a deterministic transform, "a table of data will be replaced with the same obfuscated form each time it is transformed, which ensures that connections between values (and, with structured data, records) are preserved, even across tables". That preserved relationship is referential integrity, which is exactly what a join needs, and it is the property a consistent tokenization service guarantees so joins and aggregation still run on tokenized data.
The join-preserving property depends only on same-input plus same-key producing same-output, and it has nothing to do with the compute engine. Snowflake does not know or care that a token got minted the same way over in Databricks. So if a single vault drives one HMAC over the customer id and both warehouses receive the output of that one transform, the token in Snowflake is byte-identical to the token in Databricks, and a join on the token column returns real matches.
This is the core of how DataShield's Ontology works: one customer-held vault (KMS or HSM) drives an HMAC, so a value tokenizes identically everywhere it lands. One key, one algorithm, one token. The moment you have two independent tokenizers, you have two different tokens for the same person, and the join fails before it starts.
Why do native Snowflake and Databricks masking policies break the join?
Because each platform's native control is scoped to that platform and configured on its own. They were never designed to agree with each other. Expecting them to is like expecting two teams to pick the same random password by coincidence.
Snowflake's column-level security gives you Dynamic Data Masking and External Tokenization. Per the docs, masking policies are schema-level objects, so a database and schema have to exist in Snowflake before the policy can touch a column, and the policy runs at query time everywhere that column shows up. External Tokenization lets you tokenize before load and detokenize at query time through external functions, which is the useful escape hatch, though the masking-policy plumbing stays Snowflake-specific even when the token provider is not. That policy object lives inside a Snowflake account, and it cannot govern a Databricks table.
Databricks is the mirror image. In Unity Catalog a column mask is a SQL UDF that takes the column value and hands back the original or a masked version, one mask per column, wired up with ALTER TABLE ... ALTER COLUMN ... SET MASK or an ABAC policy on governed tags. Those UDFs evaluate inside Databricks, and they cannot govern a Snowflake table.
Run both independently and you get different keys, different algorithms, different UDF logic. The same customer id lands as two different tokens, and any join across the two warehouses on that column comes back with nothing useful. Cleverer policy config on either side will not fix this. The fix is one shared keyed transform applied outside each engine, so both warehouses receive tokens that already agree. Note that External Tokenization and Databricks UDFs can both call the same external engine, which is the sanctioned way to make them agree instead of fighting.
The reverse direction works the same way, and it is worth walking once. A privileged Snowflake query hits a masking policy that fires an External Tokenization external function; that function calls your external engine, which looks the token up in the vault (or decrypts it, if you chose a reversible cipher) and returns the plaintext. A privileged Databricks query hits a Unity Catalog UDF that calls the very same engine, same vault, same lookup. Both warehouses detokenize through one authorization point, so the reverse mapping lives in exactly one place you can audit and revoke, not two places that can drift out of sync.
What does deterministic tokenization actually preserve, and what does it destroy?
Deterministic tokenization preserves equality and destroys almost every other property, so plan your pipeline around that fact and you will avoid most of the pain.
Equal plaintext yields equal token, so equi-joins, dedup, GROUP BY, and COUNT DISTINCT all still work on the token column. Order, ranges, and substrings do not survive. A LIKE '%smith%' on a tokenized name column returns nonsense, because tokenization shreds the byte structure of the plaintext. Google's deterministic transform emits either a base64 AES-SIV ciphertext (reversible with the key) or an HMAC digest (not reversible), and neither form bears any substring relationship to the input.
The honest engineering reality is that equality is the only operation you get for free. If a downstream query needs fuzzy match, prefix search, or range filtering on a sensitive field, tokenization is the wrong tool for that field, and no amount of key coordination changes that. Decide which columns are join keys and which ones need in-the-clear semantics before you tokenize, not after you have shipped and the analysts are filing bug reports. I walk through the ingest-time version of this same decision in tokenizing PII before it reaches the LLM.
Why is a deterministic token also a linkability oracle?
Determinism cuts both ways. The same property that makes tokens joinable, where equal plaintext always maps to equal token, is what turns them into a re-identification oracle.
Anyone holding the key can re-derive tokens for guessed values. Feed a dictionary of likely inputs through the transform, compare the outputs against your token column, and you have known-plaintext re-linking. Low-cardinality fields are the soft target: sex, ZIP, birth year, a country code. There are not many possible values to guess, so a key holder can enumerate all of them in seconds. This linkability framing is standard pseudonymization reasoning rather than something the Google doc itself flags, so credit it to the general de-identification literature rather than any one vendor page.
The standard defense is the key doing double duty as a secret pepper. An attacker who does not hold the key cannot enumerate anything, because the HMAC mixes in a secret that never appears in the data, so a dictionary of every sex, ZIP, and birth-year value still gets them nowhere. That is exactly why low-cardinality fields survive at all: the entropy lives in the key, not the value. Pair the pepper with per-field key separation and a single leaked key re-links one field instead of your whole low-cardinality estate. The residual risk is squarely the key holder, which is why the security of the whole scheme collapses to key custody, and why customer-held keys in KMS or HSM are treated as a primary control here rather than an optional add-on.
On algorithm choice, NIST specifies its format-preserving encryption modes FF1 and FF3 in Special Publication 800-38G, both modes of AES; FF3 was later found to have cryptanalytic weaknesses and was revised to FF3-1 in the revision SP 800-38G Rev. 1, a reminder that picking a sound keyed primitive is not optional and that rolling your own is not an answer. On compliance, GDPR treats pseudonymized data as personal data under Recital 26 and Article 4(5), so tokens are pseudonymous PII in regulatory scope rather than anonymized and out of scope. Treat them like the PII they stand in for.
What do agents and the OWASP Agentic Top 10 have to do with tokens?
Agents change the picture the moment you let one loose across both warehouses. A join-preserving token lets that agent operate over Snowflake and Databricks without ever touching raw PII. It joins, dedups, and reasons on TOK_ values, and only a separate authorized detokenization path, holding the vault key, can reverse them.
That matters because agents are a fresh attack surface, and the industry now has a list for it. The OWASP Top 10 for Agentic Applications, published December 9, 2025, is the first flagship list built specifically for autonomous agents, with categories ASI01 through ASI10. The top category is ASI01 Agent Goal Hijack, where attackers hide new goals inside documents, emails, and RAG results that the agent then treats as instructions. OWASP cites EchoLeak as an example. EchoLeak is tracked as CVE-2025-32711 and was disclosed in June 2025 at CVSS 9.3, a zero-click prompt-injection in Microsoft 365 Copilot where a single crafted email made Copilot exfiltrate internal data with no user interaction. Aim Labs responsibly disclosed it and it was patched server-side, with no confirmed exploitation in the wild; an independent academic case study later walked through the exploit chain in detail, and the authoritative severity and patch record lives in the CVE-2025-32711 advisory. The related MCP tool-poisoning class that Simon Willison documented in April 2025 has the same shape, malicious instructions tucked into a tool's description that the model reads and the user never sees, and it too is a proof of concept rather than a breach.
Tokens earn their keep in this setting. If a hijacked agent gets talked into exfiltrating your join key, it walks off with opaque TOK_ values instead of names and card numbers, which narrows the blast radius of an ASI01 goal-hijack down to pseudonyms. The Model Context Protocol security best practices (spec revision 2025-06-18) push the same direction from the identity side: MCP servers acting as OAuth proxies must implement per-client consent to avoid the confused-deputy problem, and clients must include the RFC 8707 resource parameter so each access token is bound to a specific server. Token binding and data tokenization are two halves of one instinct. Assume the agent will be manipulated, and make the thing it can reach worth as little as possible.
Best practices to prevent token leakage across warehouses
This is an engineering threat model rather than a full compliance program, so read the list as a floor to build on. Six things that actually move the needle:
- Hold your own keys in KMS or HSM, with tight custody. The whole scheme's security collapses to key custody, so this control carries the most weight. If the token provider holds the key, you have handed your linkability oracle to a vendor.
- Rotate with versioned keys, never by swapping the key underneath live tokens. Here is the trap that turns a control into an outage. Rotate a deterministic HMAC or tokenization key naively and every token changes, which silently breaks every historical cross-warehouse join the tokens existed to preserve. The fix is a key-generation id stamped alongside each token (or namespaced into it), plus a planned re-tokenization pass at cutover, so an old token and a new token for the same person are never compared as if they matched. Rotation belongs on the list, but a careless rotation is the exact failure the whole design exists to prevent.
- Separate keys per context or per field. Use a different key for the marketing domain than for the clinical domain. A leaked key then re-links one domain instead of your entire estate, and it is what makes low-cardinality fields survivable.
- Treat tokens as PII still in scope. Pseudonymized data is not anonymized data. Keep them under the same access controls, retention rules, and audit as the plaintext they represent.
- Put detokenization behind strong authz and audit. Reversing a token should be a privileged, logged, revocable action that no service inherits by default. Bind it to identity the way our auth layer binds each tool call.
- Design for equality only. Equi-join and dedup are the operations you get. If a pipeline secretly leans on substring or range behavior over a tokenized column, it fails in ways that look exactly like data-quality bugs, which is the hardest kind of failure to debug. Catch it at design time.
None of this is exotic. It mostly comes down to refusing to let convenience quietly rebuild the plaintext you worked to remove.
How do you govern the data plane, not just the agent?
Most agent-security advice stops at the prompt: guardrails, refusals, input filtering. That governs what the agent will do, and it does nothing about what the agent can reach. Both layers matter, and the data plane is the one most teams skip.
Governing the reachable surface looks like this. Tokenize sensitive fields at ingest, so a hijacked agent finds tokens where it was hoping for raw PII. Authorize per tool call, with mid-session revocation, so a session that turns hostile gets cut off mid-flight instead of at the next login. Seal every call into a tamper-evident audit chain you can verify after the fact, so an incident review becomes a matter of checking proof rather than taking anyone's word. That is the shape of the DataShield architecture, and the tokenization piece is deliberately kept outside the warehouse. One customer-held vault drives one HMAC that lands identically in Snowflake and Databricks, which is precisely why the cross-warehouse join survives.
Two honest caveats, because you should hear them from me and not stumble on them later. DataShield's join-consistency and HMAC-vault behavior described here are the product's stated capabilities, not claims you can audit from public primary sources, so weigh them accordingly. And DataShield does not have a SOC 2 report yet. If a token vault vendor gets cagey about either of those, that tells you something. If you want to pressure-test the join-consistency claim against your own two-warehouse setup, start a scoped evaluation and bring your ugliest cross-platform join.
Watch: related explainers
Background on tokenization and the agent-security context around it.
Frequently asked questions
Can I run a LIKE or substring query on a tokenized column?
No. Deterministic tokenization preserves equality and equi-joins only. It destroys the byte structure of the plaintext, so LIKE, prefix search, and range filters do not work on the token. If a field needs those operations, do not tokenize it, or keep a separate governed path for that use.
Do Snowflake and Databricks produce the same token for the same value by default?
No. Snowflake masking policies are schema-level objects scoped to a Snowflake account, and Databricks column masks are Unity Catalog UDFs scoped to Databricks. Configured independently, they use different keys and logic and will tokenize the same customer id to two different tokens, which breaks any cross-warehouse join. You need one shared keyed transform applied before both engines, or both engines calling the same external tokenization service.
Are deterministic tokens the same as GDPR anonymized data?
No. Deterministic tokens are pseudonymous data, and under GDPR (Recital 26 and Article 4(5)) pseudonymized data is still personal data in regulatory scope. Keep tokens under the same access controls, retention, and audit as the plaintext they represent. Do not treat tokenization as an off-ramp from compliance.
What happens if the tokenization key leaks?
The token becomes a re-identification oracle. Anyone with the key can push guessed values through the same transform and match the outputs to your token column, re-linking records. Low-cardinality fields like sex, ZIP, and birth year are especially exposed. That is why customer-held keys in KMS or HSM, per-context key separation, and versioned key rotation are first-class controls.
Which keyed algorithm should I use for cross-warehouse tokens?
For irreversible join keys, a keyed HMAC over the value works and is what DataShield uses. If you need format-preserving, reversible tokens, NIST specifies FF1 and FF3-1 in Special Publication 800-38G (use FF3-1, since FF3 had cryptanalytic weaknesses). Whatever you pick, the key lives in a KMS or HSM you control, because the security of the whole scheme reduces to key custody.