How irreversible Data Anonymization and a controlled Bronze-to-Gold refresh support Oportun’s FTC-related retention requirements.
This is a Databricks case study explaining how Oportun irreversibly anonymizes qualifying personally identifiable information, or PII, associated with leads, applicants, and members while preserving only the approved data characteristics required for analysis.
The same information can serve different purposes across a data estate. As part of our FTC-related retention process, qualifying PII is deleted from operational databases. In the lakehouse, deleting an entire record would also remove approved features and relationships that remain useful for statistical analysis. We needed to replace the original identity-bearing values in the active Databricks table state, preserve only the approved analytical structure, and later remove eligible obsolete files through maintenance.
We addressed that challenge by combining our Data Sanitization service with a controlled Delta Live Tables workflow. Delta Live Tables, or DLT, is now called Lakeflow pipelines. The design transforms qualifying PII in Bronze into irreversible anonymized output, rebuilds Silver and Gold from that transformed state, and relies on DLT maintenance to remove eligible obsolete files after the configured retention conditions are met.
Here’s what we’re going to cover
- Why data retention becomes a lakehouse engineering problem
- Why historical Bronze changes require controlled recomputation
- The exact four-step production workflow
- How Oportun Data Sanitization performs irreversible anonymization
- How the transformed state reaches Silver and Gold
- What we learned from productionizing the design
- Frequently asked questions
Key takeaways
- We delete qualifying PII from operational databases and irreversibly anonymize the corresponding PII in Bronze, retaining only approved analytical characteristics.
- The production workflow stops DLT, anonymizes Bronze, full-refreshes Silver and Gold, and waits for automatic VACUUM maintenance.
- Configuration-driven tagging applies one consistent transformation in Bronze, and downstream recomputation carries that state through Silver and Gold.
Why data retention becomes a lakehouse engineering problem
Operational systems and analytical systems do not use records in the same way. Operational databases support active customer and business processes. The lakehouse supports approved statistical analysis and downstream features.
Our retention process identifies qualifying PII associated with leads, applicants, and members. The specific eligibility rules are governed by policy and operating procedures, so the public architecture does not hard-code a threshold or execution cadence.
For qualifying records, the operational outcome is deletion of PII from source databases. The analytical outcome is different. Removing the complete lakehouse record would also remove non-identifying attributes, features, and relationships that remain approved for analysis.
The engineering goal was therefore precise: replace the original identity-bearing values in the active Databricks table state, preserve only the approved analytical structure, propagate that state through every downstream lakehouse layer, and later remove eligible obsolete files through maintenance.
This is an anonymization problem, not an access-control problem. For this workflow, our Data Sanitization service returns irreversible transformed output. The original PII cannot be reconstructed from that output.
Why historical Bronze changes require controlled recomputation
We organize lakehouse data through Bronze, Silver, and Gold layers. Bronze contains the Delta representation used by downstream processing. Silver turns Bronze events into refined and merged datasets. Gold provides curated data and features for approved consumption.
Normal incremental processing handles new information efficiently. The retention workflow is different because it changes records that downstream tables have already processed.
In this implementation, updates to previously processed Bronze records conflicted with the behavior of the downstream Silver streaming tables. Changing a historical value in Bronze did not by itself reconstruct every Silver or Gold result that had been derived from the earlier value.
The solution was to make the transformed Bronze state authoritative for the downstream lakehouse layers, then deliberately rebuild the derived state. We anonymize the qualifying PII once in Bronze and use a controlled full refresh to propagate those values through Silver and Gold.
The exact four-step production workflow
The order is important because each step establishes the state required by the next one.
- Stop the DLT pipelineThe pipeline is stopped before historical Bronze records are changed. This creates a controlled boundary between normal incremental processing and the retention operation.
- Anonymize PII information from Bronze tablesWe identify eligible Bronze records and filter them into a Spark DataFrame. We pass the DataFrame to our Data Sanitization service, which returns irreversible anonymized values. We then write those transformed values back to Bronze.
- Start the pipeline in full-refresh mode and wait for the anonymized data to propagate to Silver and GoldThe pipeline restarts in full-refresh mode so Silver is rebuilt from the transformed Bronze state and the result continues to Gold. Bronze is configured with
pipeline.reset.allow = false, which protects it from full recomputation while the downstream layers refresh. - Wait for VACUUM to run automatically as part of DLT maintenanceDLT maintenance runs VACUUM. Eligible obsolete files are removed after the configured retention conditions are met. Updating the active Delta table state and removing superseded physical files are distinct parts of the lifecycle.
Oportun’s controlled workflow for irreversible PII anonymization and downstream recomputation across Bronze, Silver, and Gold.
Inside Oportun Data Sanitization
Data Sanitization is the centerpiece of our adaptation. Its role in Step 2 can be understood as four operations: filter the qualifying Bronze records, call Data Sanitization, receive irreversible anonymized output, and update Bronze.
We begin by filtering eligible records into a Spark DataFrame. We pass that DataFrame to Data Sanitization with the schema and PII tag configuration. The service returns transformed values from which the original PII cannot be recovered. We then write those values back to the Bronze tables.
The service is configuration-driven. A schema describes the incoming data structure, and owners tag the sensitive field paths. Standard tags define how supported categories of PII are transformed. When supported fields are introduced, teams can extend the tag configuration without embedding field-specific transformation logic in the workflow.
The goal is to preserve only what downstream processing legitimately needs. That can include the approved data shape, supported features, and required relationships between records. It does not include a path back to the original PII.
How the transformed state reaches Silver and Gold
After Bronze is updated, the downstream full refresh re-establishes consistency across the medallion layers. Silver is rebuilt so its refined state reflects the anonymized Bronze values instead of the values processed before the retention operation.
Gold is then recomputed from the refreshed downstream state. This makes the transformed data available to approved analytics and feature pipelines without retaining the original PII in the active lakehouse state.
The pipeline.reset.allow = false setting establishes an important boundary. Bronze is not reset as part of the full refresh. It remains the transformed source for downstream lakehouse processing while Silver tables and Gold views are reconstructed.
This division of work makes the design easier to reason about. Data Sanitization performs the irreversible transformation. Full refresh propagates the transformed logical state. DLT maintenance handles eligible obsolete files later.
What we learned from productionizing the design
Apply irreversible anonymization at the source of downstream lakehouse truth
Transforming Bronze once provides one authoritative state for Silver and Gold. Independently changing every layer would duplicate logic and increase the possibility of inconsistent treatment.
Recompute derived state when historical changes conflict with streaming behavior
Incremental processing is efficient for new events. A historical mutation changes a record that downstream tables may already have consumed. A controlled full refresh lets those layers be reconstructed from the corrected source state.
Preserve approved analytical utility, not identity
A retention transformation should retain only the data shape, features, and relationships approved for analysis. It should not preserve the original PII or a reversible representation of it.
Treat logical updates and physical cleanup as separate stages
Writing anonymized values changes the active table state. VACUUM later removes eligible obsolete files after the configured retention conditions are met. Both stages matter when explaining the complete data lifecycle.
Conclusion: a repeatable Databricks data-retention pattern
The most important result was not a single anonymization function or pipeline setting. It was a repeatable operating pattern for data retention in the lakehouse.
We identify records covered by our FTC-related retention criteria, delete qualifying PII from operational databases, and irreversibly transform the corresponding PII at the Bronze source for downstream lakehouse processing. A controlled full refresh reconstructs Silver and Gold from that transformed state. DLT maintenance then removes eligible obsolete files after the configured retention conditions are met.
This design supports our retention requirements while preserving only the approved analytical characteristics the lakehouse still needs to provide. It also creates a clear separation of responsibilities among policy-based selection, irreversible anonymization, downstream recomputation, and physical file maintenance.
Frequently asked questions
