In the path, not after the fact
Scrubbing downstream is simpler to build and leaves a window in which unprotected data exists somewhere. Putting enforcement in the path removes the window at the cost of a harder latency budget.
Near-real-time privacy pipeline
A scrubbing mechanism for support data that met privacy obligations without giving up the low latency the data was valuable for.
- Organisation
- Microsoft
- Period
- 2022 – 2024
- Area
- Data platform
Context
Support data is most useful while a case is open. It also carries personal information, which brings regulatory obligations about what may be retained and in what form.
The problem
The usual resolutions are unsatisfying in opposite directions: scrub in batch and the data is stale by the time it is clean, or restrict access and the operational value disappears. Neither treats low latency and privacy enforcement as things that have to hold at the same time.
Constraints
- Processing had to keep pace with a near-real-time pipeline.
- It had to fit into the existing support-data flows instead of replacing them.
- Privacy enforcement could not be best-effort.
- Operational availability of the data had to be preserved.
My contribution
Shared ownership stated as such. Nothing here claims sole authorship of a platform.
- Architected and developed a near-real-time scrubbing mechanism for support data.
- Led development of a related data-movement capability that improved near-real-time availability of support information by removing a major dependency from the critical path.
Approach
Move enforcement into the streaming path instead of running it as a downstream batch, so data is protected as it flows instead of after it has settled somewhere unprotected.
Remove the dependency that was forcing the latency instead of optimising around it. The critical path was long because of what it passed through, not because any single step was slow.
Fit into the existing pipeline instead of standing up a parallel one, keeping a single path for the data and a single place where the guarantee holds.
Decisions that mattered
The ones with a real cost on the other side.
Outcome
Privacy enforcement applied to support data in near real time, improving regulatory posture while keeping the data available quickly enough to be operationally useful.
What generalises
- When two requirements are treated as a trade-off, it is worth checking whether the dependency forcing the trade-off is actually necessary.
- A guarantee that holds in one place is easier to keep than one asserted in several.