Data Authenticity & Lineage - Do you have full visibility of your data's journey?

Question: Can you demonstrate to others where your data came from and what has happened to it on its journey to its final destination?

Top Tip: When in the planning and design stages of a data migration or a BI/AI data preparation project always ask yourself that question. You can easily build audit fields and data inspection points in to a solution when you plan it from the start, but it is extremely difficult and sometimes impossible to go back and do this retrospectively.

Why?

If you are expecting business users to trust data it is vital that you can prove where data has come from. You may also have to report to Data Governance and Compliance teams to prove the authenticity of your data.

When you migrate data from one system to another, or implement a BI or AI Single Source of Truth Lakehouse or Data Warehouse always ensure that you have the means to trace back to source and demonstrate where data has originated from, along with any filtering or manipulation that may have been carried out along the way.

Knowing where your data has come from is extremely important for impact analysis when source systems are changing or system failures cause missing or corrupt data.

As you take your data on its transformation journey from dirty data to clean, quality published data ask yourself the following:

• Can I still access the data in its original state?

• Can I see what the data looked like at a particular point in time?

• Do I have visibility over whether a value is a stored value or a derived value?

• Can I see if a value has been changed during its transformation?

• Can I see what logic has been used to add missing data values?

• Can I provide users with details of which original data source a value has come from?

• Am I able to show the flow of data from original source to destination?

• Can I demonstrate who has access rights to data along its journey?

When planning your data solution always strive for transparency to support data governance, compliance, auditing, troubleshooting, impact analysis and the trust of data consumers.