All the hype over the last decade around Big Data suddenly seems to be coming into fruition. Everyone wants to talk about how they are getting insights from data, the value they derive from it, and how it drives decisions. Data is one of the most valuable assets for the modern-day organization.
However, the insights and decisions from data are only useful if it is high-quality. IBM estimates that the US economy is at a $3.1 trillion deficit per year due to bad data, based on the time it takes to correct the mistakes. Sloan Management Review says that knowledge workers waste up to 50% of their time dealing with mundane data quality issues, or 80% for data scientists.
A data quality check-up can help you form a pipeline that creates and sustains high-quality data from the beginning. When conducting a data quality check-up, it should consider five main criteria.
- Accuracy – the data must do as described
- Relevancy – the data should be sufficient for its intended use
- Completeness – any data should not have missing values or whole records
- Timeliness – the data must be up-to-date
- Consistency – data formats should be the same throughout all records
The standards of the requirements may vary depending on the source. For example, you may accept a higher tolerance for incompleteness from a third-party list, but the core metrics remain the same. There are six steps that a company can follow as part of a data quality check-up, ensuring best practices.
Define your goals
The first in any data quality check-up is to define your goals for improvement. Within this, you should include the stakeholders and the impact of data on business processes. For example, if the business wants to ensure all customer records are unique, they should map this to any processes it will effect, such as marketing campaigns, invoicing, or reporting.
Review existing data
Existing data will need to go through an assessment process as part of the data quality check-up. The review should account for the criteria outlined at the start of this post; accuracy, relevancy, completeness, timeliness, and consistency. In this instance, you might look at the volume of customer records that are not unique to profile the extent of the problem
The analysis phase of a data quality check-up compares the review of the existing data to the business goals. If there is a large gap between the data and where you need it to be, teams need to identify the root cause of the problem. For example, there could be a specific channel causing duplicate customer records.
Improve the data
Following the analysis, a data quality check-up process should create an improvement plan. A robust strategy includes timeframes and cost of a solution, as well as listing the stakeholders that should be involved.
Deploy the plan
The improvements that are specified from the data quality check-up can now be deployed. They should follow a standard change management process to assess any risks before implementation. All stakeholders should be aware of the reason for the change.
Once a change in place, it should become a cyclical part of the data quality check-up. In our example, the percentage of unique records should stay as a critical metric to ensure it is controlled on an ongoing basis.
A data quality check-up is a continuous process that needs the whole organization to focus on and be driven by data. With an appropriate strategy, it can ensure your data assets are valuable across all projects, helping businesses make the most of their prized possession.