The Algebra of Data Quality

Blogger: Lyn Robison

A basic algebra formula illustrates the value of data quality. To be sure, it is an illustration or a metaphor more than an actual formula for data quality, but it does demonstrate the need for and the value of information/data quality in enterprises today. The formula is a familiar one:

y = Ax² + Bx + C

In this illustration, "y" is the value that IT contributes to the business. "x" is the degree of data quality, ranging from 0 to 100. "A" is the money that the organization has put into BI systems. "B" is the money that the organization has put into operational systems. "C" is the money that the IT group spends managing complexity and building and running systems.

As you can see, if "x" (data quality) is zero, then the only value that the IT group contributes to the business comes from managing complexity and building and running systems. If x = 0, the value to the business of BI systems and operational systems (represented by "Ax²" and "Bx") are both zero.

You will also notice that when it comes to BI systems, the value is based on the square of the data quality (Ax²). If the data is of high quality, BI systems will yield highly valuable insight to the business. By contrast, if the data is of poor quality, the BI systems will produce nothing but garbage and will therefore be useless to the business.

There are many IT departments out there which are run by IT people who firmly believe that IT’s role is to manage complexity and to build and run systems for the business. In these IT organizations, data quality is not a high priority. As a result, the business derives little or no value from their BI systems and their operational systems. In these situations, the businesspeople are looking for ways to replace their IT groups, by adopting SaaS, cloud computing, and IT outsourcing. Clearly, IT people who believe that IT’s role is merely to manage complexity and to build and run systems are will find themselves competing unsuccessfully against external service providers who can do the job far more efficiently. That is not a good career path for IT people. On the other hand, data quality offers an excellent career path for IT people, because data quality is the key ingredient in the value that any IT group delivers to the business.

I recently read a fascinating article by a senior editor at Securities Industry News that illustrates how data management is no longer optional, especially for financial firms. Here is an excerpt:

… In fact, some firms may eventually decide to designate a chief data officer to manage a team of data specialists, such as data modelers, data stewards and taxonomists. The specialists, in turn, will figure out what data is stored where, verify its accuracy, build data models and create and attach metadata. The data officer and the team would decide who may access what data, integrate databases, and maintain archived data to meet regulatory requirements.

Most firms so far are maintaining the status quo -- keeping different pieces of data in many different databases.

Often, according to Atkin and other data experts, this can amount to as many as 30 different, somewhat redundant databases with discrepancies and errors, as well as 30 or more applications that consume reference data and incoming data feeds from over a dozen data vendors.

Such a hairball for collecting data just won't cut it any more.

"How a firm reacts to the need for accurate, timely, integrated data is now a critical differentiator in a competitive environment," warns Dayle Scher, a senior analyst with Tower Group, a Needham, Mass-based research shop. "Financial firms must be prepared to respond to increased due diligence on the part of investors and certainly more restrictive regulations looming."

After ensuring the data is accurate, says Scher, firms will need to share that data with downstream applications such as order management and portfolio accounting software and staff that need it, in the right formats and in a timely manner…

You can read the full article (entitled “Follow the Data When Tracking Systemic Risk”) here: http://www.information-management.com/news/data_management_risk_grc-10016449-1.html

I have discovered some excellent sources for eye-opening information on the need for data quality, including "Does Your Business Suffer From a Data Quality Reality Gap?", a YouTube "Information Management Fairy Tale", and www.IQTrainWrecks.com, a website dedicated to information/data quality disasters from around the world.