Data Management Strategies

Spinning your Wheels with BI

Blogger: Lyn Robison

BI system implementations follow a cycle. The cycle starts when businesspeople begin seeing the need for better intelligence. The decision is eventually made to implement a new BI system. The enterprise spends hundreds of thousands of dollars on BI, ETL, and DW tooling. They spin up a project or series of projects to load the data into the DWs and to create the queries and reports that deliver the intelligence to the businesspeople. With the cost of time and labor, development tools, software licenses, and computer hardware, it is quite easy for an enterprise to spend a million dollars or more on a new BI system.

The sophisticated tooling, the complex BI systems, and the cutting edge technology are impressive. It is easy to think of the BI system as a large, highly developed, powerful engine. All that is needed to fire this engine up is some fuel. That fuel, the data, comes from the enterprise’s operational systems. Powerful ETL tools extract the data from the operational systems and transform the data so that its shape conforms to the mold in the DW. The ETL processes load the data the way that barrels of oil or stacks of lumber or bushels of wheat are loaded into warehouses (after all, data is nothing but a fungible commodity that can be easily replicated, aggregated, summarized, sliced and diced, right?). Once the data is loaded, the BI engine does its work beautifully. The engine plows through huge quantities of data, aggregating it, summarizing it, analyzing it, and allowing businesspeople to mine it for gems. Sure, the data is not perfectly accurate, but everyone assumes that the numbers will be close enough. After all, smart people are accustomed to working with uncertainty. And look at that BI system do its work! The complex, powerful BI system is truly a sight to behold, a thing of beauty, and the IT people are justifiably proud of their hard work.

Then the questions start. A few businesspeople ask to see where the BI system’s numbers are coming from. Are the operational systems from which the BI data is pulled the authoritative sources for information on the topic at hand? Did data from all of the relevant data sources get included? Is there any documentation on the BI system’s data handling, such as data models or data flow diagrams that show the data’s lineage? What calculations and processes does the BI system use to produce this information? Do those calculations and processes reflect the corporation’s policies and business rules? Does the information reflect the changes that occurred when that new manufacturing plant came online or when we switched to that other supplier? Were the identifiers reconciled properly between systems? Can anyone say for sure whether or not customer XYZ and customer 47 are actually the same customer? If not, how do we know that the BI system is summarizing the data about each of these customers correctly? If the numbers from the BI system are off, how far off are they? How can we know?

These questions about the reliability of the BI information eventually become a problem. No one can give a satisfactory ROI calculation for any data quality efforts. So data quality issues are never addressed in a systematic way, and as a result, the data quality is consistently neglected and continually deteriorates. Eventually, businesspeople aren’t sure how much they can trust the numbers coming from the new BI system. Businesspeople begin seeing the need for better intelligence. The decision is made to implement a new BI system. Here we go again. Let’s build another BI system!

In this cycle, the enterprise tries in vain to address problems of information reliability and usefulness by creating yet another system. Unfortunately, that does not solve the enterprise’s basic problem. The enterprise does not lack systems. The enterprise lacks reliable, useful business information.

Eventually, the business leadership sees that they are spending millions of dollars on IT, and they begin to look for cheaper ways to get business intelligence, through approaches like IT outsourcing, cloud, and SaaS. If they could peer into the future, they would see that IT externalization will at best give the business the same mess for less.

The fundamental problems that make BI systems ineffective are the mistaken perceptions about data that are harbored by most IT professionals. In truth, data is not merely a fungible commodity that can be moved around and aggregated and summarized to fill up data warehouses the way that crude oil can fill up oil tanks or steel barrels. To be useful for any purpose, data must be an accurate picture of reality.

To be an accurate picture of reality, data must accurately represent individual, non-fungible assets. Barrels of oil might be fungible (interchangeable) – one barrel of oil might be just as good as another, but one customer is not just as good as another. Customers are not interchangeable. Neither are buildings, factories, documents, contracts, decisions, or any number of vital business assets. If the data representing those non-fungible assets is treated merely as a fungible commodity, the data will not accurately represent reality, and the enterprise’s data will be expensive but unreliable for business decisions.

Leverage Data from disparate Silos for Configuration and Change Mgmt

Blogger: Lyn Robison

I recently presented a case study in how a large enterprise is successfully leveraging data from disparate silos for CMBD and change management.

You can view the presentation here:

Meatloaf with Mashup Potatoes

Blogger: Joe Maguire

I like out-of-the-way restaurants in sleepy little hamlets.  The food might not constitute haute cuisine, but the quaint, low-tech atmosphere offers a stark, refreshing contrast to the circles I usually travel in.  How nice it is to peruse a non-drop-down menu.

But I spend so much time thinking about technology, it takes more than a trip to Sap Bucket, Vermont or Tumbleweed, Oklahoma to distract me.  Last time I scanned a dog-eared, laminated menu in a roadside diner, I couldn’t help noticing that I was encountering a mashup. The menu contained four different kinds of data, conforming to four different meta-models:

• Page one showed a reproduction of a pen-and-ink drawing of the diner, as it looked 70 years ago.  (Meta-model: Image data.)

• Page two contained a charming (and apocryphal) history of the diner, founded by great-grandmother as a venue for her cherry pie, a perennial prize winner at the county fair and a favorite of barn raisings and church bake sales for generations.  (Meta-model: Narrative data.)

• Page three contained the menu, with categories for breakfast, lunch, dinner, and dessert.  Certain items were flagged as vegetarian, others as “available in children’s portions.”  (Meta-model: Structured data.)

• Page four contained a map of the area, with “Grandma’s Diner” prominently marked.  (Meta-model:  Geospatial.)

There was no evidence that any computer technology was used to prepare the menu.

The diner menu shows that humans are perfectly comfortable switching between meta-models.  Even the most famished diner knows—instinctively, automatically—that the narrative data should be read sequentially, whereas the structured data can be scanned in any arbitrary order.  Life is short; choose dessert first.

This is yet another case where wetware (i.e., human intellectual capacity) outshines software.  Getting software to cope with multiple meta-models is tricky.  Exhibit A:  the profound difference between query and search.  Exhibit B, in the form of a question: Why, after it took the computer industry so long to develop mashup techniques, did end users accepted mashups almost immediately?

Users almost never discuss meta-models because they don’t need to.  Computer scientists, on the other hand, do quite a bit of lip-flapping about meta-models.  This disparity can lead to an erroneous conclusion: That meta-models are the exclusive province of computer scientists.  This is a serious mistake, with consequences for requirements analysts who interact directly with users.

The lesson here is this:  Computer scientists must know that users employ meta-models, and must learn to distinguish between meta-models that arose organically as a by-product of human culture and meta-models that were concocted by computer scientists to accommodate technological discussions.  Call them organic meta-models and invented meta-models.

Organic meta-models are legitimate vehicles for gathering requirements from users.  Invented meta-models are not.  For data modeling, the most popular invented meta-model in use today is the UML class diagram.  UML class diagrams are based on the object-oriented paradigm, which did not arise organically within human culture as a way to organize a certain kind of data.  Rather, object-orientation was explicitly designed by computer scientists to promote certain desirable traits in software, including reusability of programming artifacts.

Because it is based on an invented, not organic, meta-model, UML class diagramming is not an appropriate vehicle for requirements gathering with users.   That’s why in my decades-long career as a data modeler and requirements analyst, I have never shown an end user a UML class diagram.

Data Warehouses and non-fungible Assets: a bad Combination

Blogger: Lyn Robison

If you do a Google search on "data warehouse failures", you will get a taste of the hazards of data warehousing. One sure path to failure is to try to do something with a data warehouse that data warehouses are not good at. That would be using a data warehouse to track or manage non-fungible assets.

Fungibility might sound like an obscure concept that has little to do with implementing successful information systems, but fungibility is central to relationships, collaboration, ILM, information security, identity management, and business intelligence. And non-fungible assets are something that data warehouses don't manage well.

Understanding the concept of fungibility can enable IT professionals to avoid a common path to failure when implementing a broad range of information systems. A lack of understanding of fungibility is one reason why data warehouses and business intelligence projects often don’t deliver their promised benefits, why enterprises have had such a difficult time implementing CRM systems, and why few if any large enterprises know how many customers they have. If you don’t know how many customers you have and who they are, you can’t know how much each customer is buying from you, or understand what each customer is asking for, or manage how each customer relates to other customers or to your company’s salespeople. And it’s not just customers. If your information systems cannot give you an accurate picture of the non-fungible assets owned by a typical enterprise, you can’t manage them and you can’t manage anything that relates to them.

Q. What is fungibility?
A. You can find a good definition of fungibility here: http://en.wikipedia.org/wiki/Fungibility. You can also find some thoughts on IT fungibility here: http://it.toolbox.com/blogs/minimalit/fungibility-27324. A few quick examples would be: ¾” sheet metal screws are fungible, kiln-dried douglas fir 8’ 2x4s are fungible, barrels of oil of the same type and grade are fungible, dollar bills are fungible, but buildings are not fungible, employees are not fungible, and customers are not fungible. Fungible commodities have attributes and characteristics, with type and quantity being the most important considerations. Non-fungible assets have identity (in addition to attributes, characteristics, and type), with identity being the most important consideration.

Q. What happens if I treat fungible commodities and non-fungible assets the same way?
A. You will not achieve the results you want. (Click on the Dilbert cartoon below to see an example.)

 

Q. What happens if I treat data about fungible assets and non-fungible assets the same way?
A. Your information systems that are intended to track non-fungible assets will not be successful. Examples abound. Enterprises cannot easily determine which data to archive in an ILM setting because they don’t know which data refers to long-lived, non-fungible assets. A large, multi-national non-profit organization does not know how many buildings it owns. The most sophisticated army in the world does not know how many battle tanks it has. Our friend and colleague Joe Bugajski very nearly died when medical professionals could not keep track of his medical condition and treatment (see Joe’s blog post and my related post).

Q. What is the worst way to treat data about non-fungible assets?
A. In two words, “data warehouses”. Data warehouses are best-suited for aggregating, calculating, and analyzing quantities of fungible commodities. For example, they work well for adding up the dollars of revenue and expense for an income statement. This is because data warehouses shine when working on type and quantity. Unfortunately, data warehouses are often misapplied for use with data about non-fungible assets, and millions of dollars of sad experience have shown that data warehouses are a manifestly poor platform for managing identity, as a Google search on "data warehouse failures" will reveal.

Q. What is the best way to treat data about non-fungible assets?
A. In a word, “registries” (another word might be “directories”). Identity is the most important consideration for managing non-fungible assets. For guidance on registries and identity, see "Delivering Integrated Information from Data Silos Using MODS" and the Identity and Privacy home page.

Gartner buys Burton Group

Blogger: Lyn Robison

Burton Group CEO Jamie Lewis called together the BG Research Directors and Service Directors for a series of meetings this week at HQ in Salt Lake City. As the first meeting began on Monday, Jamie stepped to the front of the room to present a set of slides, which was not unusual. But then, one of the first slides he showed had a big Gartner logo right in the middle. Jamie then said, as near as I can recall, “Burton Group has been purchased by Gartner.”

I was so shocked that I almost could not process what I was seeing and hearing. After the initial surprise and after hearing the rationale behind this acquisition, I began to see the potential of this combination. Burton Group’s depth with Gartner’s wide distribution should be a great combination.

At that Monday meeting, the Burton Group Research Directors and Service Directors asked lots of pointed questions of the Gartner leadership, which included Gartner CEO Gene Hall and Gartner SVP of Research Peter Sondegaard. Our questions centered on how this acquisition by Gartner would affect Burton Group analysts’ ability to give our Burton Group clients the fiercely independent and technically thorough guidance to which they have become accustomed. I’ve got to say, I liked what I heard from the Gartner executive leadership at that meeting. Gartner bought Burton Group because they like the guidance that we provide, and they want us to continue to provide that guidance to our clients and to a whole bunch of additional clients in the future.