Deprecation for Data Engineer \ Analyst

Data engineer and analyst walkthrough — using ODD lineage and ownership to deprecate data objects safely, informing stakeholders, and mitigating downstream pipeline failure risk.

Key words: divestiture, communication process, identifying stakeholders.

Challenge

As a Data Engineer, I initiate a deprecation process and have to deprecate my data objects mitigating the risks of other pipelines failing.

As a Data Analyst, I have found an outdated data object on books catalogue that my team has inherited from a legacy team. I know that it is of a poor quality and takes significant resources of my team to maintain it. I want to deprecate it and can't predict the consequences.

Solution

The ODD Platform provides a lineage diagram to examine data flows and the ability to identify data owners so the deprecation process goes smoothly. It also has a built-in deprecation status that automates the "retire now, delete on a future date" step the scenario below performs by hand: set a data entity's status to DEPRECATED with a switch date, and a background job flips it to DELETED automatically when that date arrives. You can drive the whole timeline in the platform instead of tracking it in a calendar — see the note after the scenario.

Scenario

1. My team has inherited a data object from a legacy team describing books (ISBN, code, author, publisher).

2. We examined the object and found out that it has not been updated for almost a year due to pipeline failovers and therefore does not contain recent book releases. I have strong doubts that expenses on storage, calculation and team support can be justified.

3. I need to ensure that divestiture won’t impose downstream system failure or block my users: - As a Data analyst, I log in to the ODD Platform to find the stakeholders, SMEs or primary PoCs of the object in doubt. - As a Data Engineer, I explore a lineage diagram to check whether there are any downstream systems sourcing from the object in doubt (pipelines, dashboards, views).

4. I find out that though there is a dashboard sourcing from my object, it was not used for 6 months.

5. We decide to hold a session with stakeholders listed in ODD to double-check my findings and notify them about divestiture.

6. We send out a notification letter that this data object is going to be decommissioned in 3 months.

7. We archive the object and stop object increments.

8. After 3 months we delete the object, DAGs and archive the code base.

Result: Deprecation process is well-managed, all users are notified in advance and risks mitigated.

Using ODD's built-in deprecation

The scenario above tracks the decommission timeline by hand. ODD can run that timeline for you:

• Mark the object DEPRECATED with a switch date. Setting a data entity's status to DEPRECATED lets you attach a switch time — the moment the object should be retired. A background job runs on a fixed schedule and automatically moves any DEPRECATED entity whose switch time has passed to DELETED. So "decommission in 3 months" becomes: set DEPRECATED with a switch date 3 months out, and the platform handles the flip — no calendar reminder required.

• DELETED is a soft delete with a retention window — not a permanent archive. Moving an object to DELETED hides it from the catalog and detaches its lineage and group memberships, but the record is retained at first — so you can restore it (move it back to an active status) and its lineage and group relationships return. It is not a permanent archive, though: the platform is designed to hard-delete a DELETED entity once it has been DELETED longer than a configurable retention window (housekeeping.ttl.data_entity_delete_days, default 30 days), cascading the deletion to its lineage, metadata, alerts, attached files (including objects in S3/MinIO storage), and group relations — with no restore path afterward. Treat DELETED as "scheduled for permanent deletion, restorable only within the retention window," not as a safe long-term archive — on a default install the purge fires 30 days after deletion. See Data Entity Statuses → The soft-delete TTL for the exact retention behaviour and how to configure the window. (The destruction of DAGs and code base in step 8 happens in your own systems, not inside ODD.)

Blast-radius check before you deprecate. Lineage reads are not owner-scoped, so the downstream graph shows consumers across the whole catalog — exactly what you want when retiring an inherited object. To catch deeper chains (a view that feeds another view that feeds a dashboard), raise the depth control rather than trusting the default one-hop view. See Data Objects Lineage for how depth and graph traversal behave.