PrivacyGate
Masks personal information in five languages, tested on synthetic data.
Overview
PrivacyGate is an offline prototype that masks personal information in English, German, French, Italian and Spanish. It combines mBERT region models with deterministic rules, so the models propose regions and the rules validate structured values. The best setup runs region-v5 and region-v6 together and masks a token when either one marks it. All work uses synthetic data; it is a research prototype, not a privacy control, and it gives no privacy or legal guarantee.
What I built
- A staged pipeline that keeps every candidate span in the coordinates of the original text.
- A context gate that separates personal from operational uses, plus address and name assemblers that extend regions and propagate name mentions within a document.
- A design that blocks the result when a required stage or inference fails, instead of silently falling back to regex.
- Blind evaluation bound to file hashes, with custody receipts, so each model arm is measured once, on test sets I did not generate.
- A Colab training bundle with hash checks, used to train region-v6 on a T4 GPU.
Pipeline
Six stages in order: mBERT region model, regex and structured detectors, context gate, address and name assemblers, refinement and coverage union, render.
- 1mBERT region modelProposes regions
- 2Regex + structured detectorsRules validate values
- 3Context gatePersonal vs. operational
- 4Address / name assemblersExtend regions
- 5Refinement + coverage unionCombine accepted spans
- 6RenderPlaceholders
Try the masking
Input
Hello, my name is Laura Becker and you can reach me at +41 79 555 01 23. Please transfer the refund to IBAN CH93 0076 2011 6238 5295 7 and send the receipt to laura.becker@example.com. I live at Seestrasse 12, 8002 Zürich.
Output
Hello, my name is [PERSONNAME] and you can reach me at [TELEPHONENUM]. Please transfer the refund to IBAN [IBAN] and send the receipt to [EMAIL]. I live at [ADDRESS].
Input
Guten Tag, ich heisse Jonas Meier, meine Telefonnummer ist +41 76 444 12 34. Bitte überweisen Sie den Betrag auf IBAN DE89 3704 0044 0532 0130 00. Meine Adresse lautet Bahnhofstrasse 5, 3011 Bern.
Output
Guten Tag, ich heisse [PERSONNAME], meine Telefonnummer ist [TELEPHONENUM]. Bitte überweisen Sie den Betrag auf IBAN [IBAN]. Meine Adresse lautet [ADDRESS].
Real output of the full pipeline on synthetic input, using the region-v5 + region-v6 union with name threshold 0.1.
Results
The headline is two independent synthetic test sets: Nemotron-PII (5,000 English documents) and ai4privacy (1,000 rows each in English, German, French, Italian and Spanish). No model trained on them, and each arm was measured once, after its options were chosen on development data. Covered is the share of personal letters and digits that end up masked. Excess is the share of unannotated characters that were masked anyway. Exposed counts the documents (Nemotron) or rows (ai4privacy) that still show at least one personal letter or digit.
The best arm is the union of region-v5 and region-v6: a token is masked when either model marks it, with name threshold 0.1. It covers 98.99% on Nemotron with 1.41% excess and 246 documents exposed, and 93.75% on ai4privacy with 2.84% excess and 568 rows exposed. Region-v5 with the step-1 options leaves 545 and 1,006 exposed. Region-v6 with threshold 0.1 leaves 1,308 and 976.
| Arm | Covered | Excess | Exposed |
|---|---|---|---|
| Nemotron-PII, 5,000 documents | |||
| region-v5 + step-1 | 96.91% | 1.37% | 545 |
| region-v6 + threshold 0.1 | 92.03% | 0.28% | 1,308 |
| v5 + v6 union, threshold 0.1 | 98.99% | 1.41% | 246 |
| ai4privacy, 5 × 1,000 rows | |||
| region-v5 + step-1 | 86.55% | 2.60% | 1,006 |
| region-v6 + threshold 0.1 | 88.77% | 1.60% | 976 |
| v5 + v6 union, threshold 0.1 | 93.75% | 2.84% | 568 |
Documents with exposed personal characters
Exposed documents on Nemotron-PII and exposed rows on ai4privacy, lower is better. region-v5 with step-1: Nemotron 545, ai4privacy 1,006. region-v6 with threshold 0.1: Nemotron 1,308, ai4privacy 976. Union of region-v5 and region-v6 with threshold 0.1: Nemotron 246, ai4privacy 568.
Lower is better. Both sets share one scale. Nemotron counts documents, ai4privacy counts rows.
Covered against excess
Covered percent and excess percent for each arm. Nemotron: region-v5 with step-1 covers 96.91% with 1.37% excess; region-v6 with threshold 0.1 covers 92.03% with 0.28% excess; the union covers 98.99% with 1.41% excess. ai4privacy: region-v5 with step-1 covers 86.55% with 2.60% excess; region-v6 with threshold 0.1 covers 88.77% with 1.60% excess; the union covers 93.75% with 2.84% excess.
Each point is one arm. The vertical axis starts at 84%. The union gains coverage at about the excess of region-v5, while region-v6 alone has the least excess.
Region-v6 on its own
I trained region-v6 on a Colab T4 GPU on train-v6, which has 47,279 rows. On its own it lowered excess, to 0.28% on Nemotron against 1.37% for region-v5 with step-1, but it missed more there (92.03% against 96.91%). It also regressed on the v7 stress layouts: 4,747 exposed letters and digits against 334 for region-v5. The union brings that back to 306. Gretel TEST is not evidence for region-v6, because train-v6 contains 23,279 Gretel train rows from the same generator.
Checksum ID detectors
I kept two detectors that validate a checksum: the Italian codice fiscale and the French NIR. The other detectors (Spanish DNI and NIE, German tax ID, British NINO, Swiss AHV, US SSN) failed the development gate or had no development evidence, so they stay off. With the two kept detectors, Nemotron and ai4privacy results did not change at any reported digit.
Remaining misses
- Single word names without a cue word, the largest miss on both sets.
- ID codes that start with letters and have no checksum, such as Nemotron personal references and ai4privacy ID card, driver licence and passport numbers.
- Ages written as bare numbers.
- Address fragments inside short sentences, such as a street name and house number on their own.
Internal sets overstate quality
The earlier internal blind sets looked better than outside text. On the v7 internal set, region-v4 with the round 4 rules covered 99.71% of personal letters and digits, but on the external Gretel finance test (4,696 documents) region-v4 covered 87.09% and region-v5 covered 88.71%.
Internal against external coverage
Personal letters and digits covered. region-v4: internal v7 blind set 99.71%, external Gretel test 87.09%, a gap of 12.62 points. region-v5: internal v7 99.13%, external Gretel test 88.71%, a gap of 10.42 points.
Both bars share a 0 to 100% scale.
Limits
- Both test sets are synthetic, so the numbers say nothing about real personal data. I mapped their labels to my scope myself, for example medical record numbers count as personal references.
- The union masks much of the text the labels call out of scope: 48.0% of those characters on Nemotron (URLs, company names, dates), against 6.1% for region-v6. These characters are not scored as excess.
- An ok status certifies that processing finished, not that every personal value was masked. 246 Nemotron documents and 568 ai4privacy rows still show some.