In the digital world and with the rise of Big Data, it has never been more important for a business to keep on top of their data quality. Whether it be consumer behaviour, predictive analytics, product information or even compliance and risk related tasks, companies need to store and analyze vast amounts of data every single day just to keep up with the competition in their industry.
Some companies are now building entire business models around their data to treat as a true financial asset and ensure they make the most out of its value strategically. Where that is the case, dirty or unclean data is simply not an option given the potential cost of managing it. To ensure data quality, businesses are introducing key metrics for its management and integrating these into every area of the business. This article details some of the key data quality metrics being used today.
Data quality metrics
First of all, it should be said that there is no such thing as a “one size fits all” strategy for data quality metrics. Ultimately it depends on your data and your business and one of the biggest myths about data quality is that is needs to ensure everything is 100% error free. With so much digital data coming in, 100% would be almost impossible and most businesses will appoint a Data Steward who owns the acceptable tolerance levels.
- Errors against volume
Typically, data quality errors will be measured against the volume of data coming into the business. If you are getting 100 errors per day but taking in over 10,000,000 data points, then it perhaps isn’t so bad but if that were 100 errors against just 1,000 records that probably signifies some sort of problem. There are usually 3 different ways to handle errors to ensure the data quality metric makes sense.
- Ignore the error – in an address if First Ave is flagged as an error vs First Avenue, does this really matter to the wider business? If not, set a rule to ignore it
- Reject the error – sometimes, if the data is heavily corrupted, it is safer to just delete the record altogether (this should be measured as a different metric)
- Correct the error – items that need correcting are ones you should be measuring. If amendments are required to the data, it must be business critical
The fewer error corrections that you require, the better your data quality.
- Data quality dimensions
There are slight variations on this but there are a few key data quality metrics that every business should be measuring. In no particular order, these are:
- Completeness – the percentage of records that contain all critical fields e.g. name and contact details are all populated perhaps. It maybe worth to first identify all the mandatory fields for your business.
- Uniqueness – data is not duplicated in your database. This can be easily measured by looking at repeated addresses, phone numbers and emails.
- Timeliness – is everything working in real-time e.g. if a customer places an order today, does your system show the correct date?
- Outdated – as time goes on, your database maybe historical and may need to be updated. B2B records may become obsolete every 2 years as people switch jobs, versus B2C records can become invalid after 5-7 years as people moves residences.
- Accuracy – does the data make sense in the context of what it is supposed to be used for, are the records up to date? This maybe impossible to measure, however, one simple fix is to put a flag on records that are outdated or incorrect or “no longer there” for your customer service, sales and marketing teams to check of.
- Consistency – does everything follow a standard pattern e.g. all dates are dd/mm/yyyy or are fields standardized/normalized.
These dimensions are just guidelines and each of those would have its own set of rules that make it work for a business as what accuracy means in one company, might be different to another for example.
The future of data quality
Whilst there is no one size fits all model for data quality metrics, the dimensions given in this article can help start to create a basic framework for what need to be measured. Data is only going to continue to grow for the foreseeable future and unless these issues are tackled now, it could end up being too late.
Now is the right time to come up with the data quality rules that will make your business a success and invest in the right data quality tools and dashboard to monitor the metrics. Often, these data quality dashboards can be created right inside your Salesforce, or marketing automation tool. If you need help with setting up data quality metrics or setting up dashboards, StrategicDB can help, we are a data cleansing and analytics company!