Data Management Strategies

So you want good data - do you know what it cost?

By Noreen Kendle

No one ever says they want bad data, everyone wants good data. The poor state of the data and lack of formal data management in most organizations is well known. Improving the state of data has been on most C-level radar for at least ten years. So what‘s the problem?  Am I missing something here?  If most organizations are aware of their data problems and they know they need good data, then why hasn’t data management been given the spotlight and funding it needs. Many of my data management colleagues say that they continually struggle to sell a data management function.  Try as they will, they cannot convince management to spend the money and resources; same folks who want good data.

I believe there is a fact that is often overlooked. If you have data, then you manage it, weather it’s recognized or not – data will get managed, proactively or reactively.  Most organizations manage their data reactively. The cost of reactive data management typically goes unrecognized and unaccounted for. I was talking to a fried the other day about data quality problems. She had been the head of a large vendor support/management department (126 employees) for a fortune 100 company. She calculated that on average 60% of their time was spent chasing and correcting bad vendor and product data. It was just considered to be part of the job, and one of the main reasons for such a large department. I have been told and witnessed similar situations across many organizations. Reactive type data management is never accounted for in a data management budget.

Request for funds for a formal “proactive” data management function is viewed as a new expenditure that requires extensive justification and is rarely approved because the organization is functioning "fine" without it. The irony is that the expenditure for a formal data management function is a drop in the bucket to what is being wasted in reactive data management.  

The case for formal data management needs to be presented as a cost-cutting initiative with an accounting and comparison to the cost for reactive data management. Proactively managing data is the key to having good data and will cost much less than most organizations are spending today. If the true cost of reactive data management were recognized,I believe formal data management would quickly take center stage in most organizations.

Is Data Integration Working?

 By Noreen Kendle

The typically approach to data integration has been a bottom-up, data systems, tools/technology approach where a data integration tool is used to merge data columns form various database systems.  This approach has been around for many years.  If it worked, then we would not be in the data disparity mess we are in today. 

The data integration tools and the automation they offer are not the point of failure; the issue is the methodology, how the tools are applied. The tools move and merge the data as directed. It is how they are directed. The decisions on what gets “merged” or “pasted” together are typically based on data field names, content and sometimes relationships within the existing data systems.  Anyone who has worked in the guts of data knows how misleading a column name or even its content is to its meaning. Add to that the fact that most legacy data systems are riddled with data issues and idiosyncrasies.  When they are integrated in this manner, the data abnormalities multiply.

There is also the dimension of time that plays a role in data meaning and integration.  Often data can change its meaning over time as can the business it represents. Even when two data items appear the same, they may be representations of the same thing at different points in time and therefore, not really the same thing. This is a challenge for data systems tool based integration.

Another challenge with this data integration methodology is capturing the data relationships. Data meaning is dependent on it relationships with other data, where the meaning of a data item, a column in a database, is relative to its relationship within the table it resides, as well as its relationship/dependencies to other attributes and tables. When data is looked at individually, as a single column or attribute, void of its relationships, there is a high risk of losing its full meaning, thus compromising its integrity. Many organization’s database systems today, even if they have the capability, do not implement referential integrity (RI) within the database or compensate for the RI.  Much of the data, even well named and defined data has lost its relative meaning. 

Data is not something that can be disassembled and treated linearly, without the risk of compromising its integrity. There is a very good reason why relational logic for data structures works well – data, as is the real world it represents is inter-related. Hierarchies are simple parent-child relationships. This does not necessarily mean data has to be stored or moved in a relational structure; rather its relational or relative meaning must be preserved in some manner in order to maintain its integrity.

Data integration tools can only take a “best guess” in an attempt to derive a data field’s meaning, even if the guess is based on a good quality name, content, and structure/relationships. The real meaning/understanding of data comes from its intended business meaning, how it represents the business.  Without the real business meaning, where the physical data is mapped/justified to a business data model and then used to drive the data integration, it can only ever be a best guess, with or without a tool. 

Is Data Management an Information Technology (IT) Function?

By Noreen Kendle

A few years ago I would not have questioned data managements fit as an IT function.  As data continues to gain recognition through initiatives such as master data management, service oriented architecture and business intelligence, IT departments have been challenged with understanding, managing, and governing the data. Many of my fellow data professionals who have been involved with data ownership/governance initiatives have expressed the difficulty of getting the business to take ownership of the data.  This is one of several reasons I have begun to question ITs role in data management. Why is it ITs job to “get” the business to own what they already own? There is something backwards about that.

Data is a representation of the organization.  The organization uses this representation, the data to operate record, manage, report and plan.  Organizations have been creating and using data long before computers were ever thought of. Data is clearly a business asset, not an IT asset as is hardware or software.  Prior to computerization, the business owned, managed, understood and governed their data assets.

Information technology is about effectively applying technology to the organizations data/information assets in order to help the organization reach its goals. Just as machine technology can be applied to a manufacturing process, information technology can be applied to a business process. The products that flow through the machine technology are never considered part of that technology or the responsibility of the machine technologist.  Products are considered a business owned asset.  So why has data and its responsibilities an IT function?    Data is clearly a business asset and most likely one of their most important assets.   It’s becoming clear to me that the responsibilities for data belong in a business department, not IT. The functions involving applying technology to the data asset, such as a DBMS, should remain the responsibility of an IT department.

The Seven Top Data Delusions

By Noreen Kendle

The world of data is full of delusions - false beliefs or ideas about data. These are fueled by the mountains of data related white papers, articles, blogs, and marketing material. If I "google" any data topic, like master data or BI, millions of hits are returned. As I skim through these, nearly all are regurgitations of the last – thus the data delusions continue to grow. It is interesting how much is assumed to be true if we read it in print.

Below are the seven most popular I continue to see:

Data Delusion One

: If the data is there then it must have been deemed good data. There are not secret data police monitoring the data in most organizations. A large percentage of incorrect data lives within the data stores.

Data Delusion Two

: If it looks right then it must be. Typically, data is considered "poor quality" when it obviously looks incorrect or is known to be incorrect. Often data can "look" right, when it is not. How do you know if the answer returned when you ask a question, using a computer system, is correct - you would not need to ask if you knew the correct answer?

Data Delusion Three

: A new tool/technology will fix the data problems. There continues to be a belief that the tools/technology will auto-magically figure out if the data is correct or belongs together. Unfortunately success is always dependant on the quality of what goes in– garbage in, garbage out is still true.

Data Delusion Four

: Data is a computer phenomenon like software or hardware. Many of the definitions support this, but data has existed for longer than before computers were ever imagined. Data is a representation of the real-world organization, its things, people, locations and events. Computers help to automate the processing of data.

Data Delusion Five

: "Cleaning" the data fixes it. There is always a reason data becomes corrupted. It just does not magically happen. Data errors or poor quality data are a symptom of a problem, rarely the problem itself. Fixing a symptom does not fix the problem - it’s like taking an aspirin for a brain tumor.

Data Delusion Six

: The data meaning can be deduced from its name/definition. Even in the rare case when a data store has been diligently modeled from a business standpoint and implemented accordingly, the data system deteriorates over time. Many of the data stores in our organizations have never been designed / modeled in the first place. The data field names and sparse definitions were often the best guess by the programmer at the time. `

Data Delusion Seven

: Data can be managed/integrate/cleaned at an individual attributes/columns level. The data attributes/ columns are intended for description purposes. They are relative to what they are describing, as well as to the relationships/ dependencies of the things they are describing. When data attributes/columns are taken out of this context and treated indiviually, they can lose much of their meaning, and thus integrity.