Data Management Strategies

The Seven Top Data Delusions

The world of data is full of delusions - false beliefs or ideas about data. These are fueled by the mountains of data related white papers, articles, blogs, and marketing material. If I "google" any data topic, like master data or BI, millions of hits are returned. As I skim through these, nearly all are regurgitations of the last – thus the data delusions continue to grow. It is interesting how much is assumed to be true if we read it in print.

Below are the seven most popular I continue to see:

Data Delusion One

: If the data is there then it must have been deemed good data. There are not secret data police monitoring the data in most organizations. A large percentage of incorrect data lives within the data stores.

Data Delusion Two

: If it looks right then it must be. Typically, data is considered "poor quality" when it obviously looks incorrect or is known to be incorrect. Often data can "look" right, when it is not. How do you know if the answer returned when you ask a question, using a computer system, is correct - you would not need to ask if you knew the correct answer?

Data Delusion Three

: A new tool/technology will fix the data problems. There continues to be a belief that the tools/technology will auto-magically figure out if the data is correct or belongs together. Unfortunately success is always dependant on the quality of what goes in– garbage in, garbage out is still true.

Data Delusion Four

: Data is a computer phenomenon like software or hardware. Many of the definitions support this, but data has existed for longer than before computers were ever imagined. Data is a representation of the real-world organization, its things, people, locations and events. Computers help to automate the processing of data.

Data Delusion Five

: "Cleaning" the data fixes it. There is always a reason data becomes corrupted. It just does not magically happen. Data errors or poor quality data are a symptom of a problem, rarely the problem itself. Fixing a symptom does not fix the problem - it’s like taking an aspirin for a brain tumor.

Data Delusion Six

: The data meaning can be deduced from its name/definition. Even in the rare case when a data store has been diligently modeled from a business standpoint and implemented accordingly, the data system deteriorates over time. Many of the data stores in our organizations have never been designed / modeled in the first place. The data field names and sparse definitions were often the best guess by the programmer at the time. `

Data Delusion Seven

: Data can be managed/integrate/cleaned at an individual attributes/columns level. The data attributes/ columns are intended for description purposes. They are relative to what they are describing, as well as to the relationships/ dependencies of the things they are describing. When data attributes/columns are taken out of this context and treated indiviually, they can lose much of their meaning, and thus integrity.

The Business Intelligence Investment: Context

Blogger: Marcus Collins
In my recent post I detailed the critical success factors for a successful analysis initiative namely:

• Having the right analysis process
• Having access to the right information
• Having the right context
• Having the ability to make decisions at the right time
• Having the right leadership
• Having the right team dynamics
• Having the right people

And finally:

• Choosing the right problem

In this post I’m going to explore the “right context” in more detail. Whilst it is true that “All men are created equal”, the same cannot be said of analysis problems. It is important to determine the context of the problem as this brings with it a set of constraints and implicit assumptions that the analyst must understand and consider as they perform the analysis.The key elements of decision context are the:

• scope
• complexity
• timeliness

Scope – is it an operational decision for example analyzing customer purchases to determine customers with the potential profit; simulate supply chains to reduce overall inventory levels and order-to-delivery times? Or is it strategic in nature for example deciding whether to move into a new market or embark on a strategic acquisition? Strategic and operational decisions have a different set of characteristics:Strategic Decisions

• Long Term
• Historic data
• Internal and External perspective
• Enterprise-wide data focus (information politics)
• Focus on analytics and interpretation/heuristics
• Poor data/information quality medium/high impact on decision
• Long term feedback loop – enterprise’s goals/mission statement

Operational Decisions

• Short Term
• Real-time data
• Internal perspective
• Business unit or function data focus
• Focus on analytics; less focus on interpretation
• Poor data/information quality – high impact on decision
• Short term feedback loop – focus on operational metrics

Complexity – problem can increase in complexity quickly and expand the scope of an analysis that is quantitative and focused on operational data to one that relies on the judgment of the analyst. The key is to be cognizant that the analytics guides the decision making.

Take this sales forecast. The data shows that Q1/Q2 has seen steady growth. A simple “interpretation” might assume that Q3 would also show steady growth. But let’s assume that Q3 shows a marked drop in sales in – will Q4 follow Path A (a return to Q1/Q2 growth) or Path B (a continuation of Q3’s falling sales). To answer this question may require data outside that which is normally available from operational systems. For example, if this was a weather dependent product or service you may need weather data to see if there’s a correlation between sales and temperature. A more complex correlation might be an increase in fuel costs leading to a drop in toll road usage. In both examples the decision required an analysis of the operational data and judgment/intuition on the part of the analyst.The final aspect of context is the Timeliness of the data. I explored this aspect in this blog post – analysis when the business needs it. The traditional view of business intelligence is of operational systems feeding a data warehouse through an Extract, Transform and Load (ETL) data pipeline. Little has changed with this model since it was initially proposed in the 1980’s. Increasingly organizations are realizing that the retrospective view of data this model supports is not sufficient to meet the demands of companies that need to function at internet speed. Rather than following this one speed approach organizations should adopt a tiered approach tailored to the organization’s business requirements, competitive environment and customer demands.

In a document to be published in early June and a series of TeleBriefings on June 2 and 3, Burton Group Senior Analyst Marcus Collins will explore the analysis context and each of the other critical success factors in more detail and provide guidance on how organizations can develop a roadmap for the successful deployment of a fact-based decision making culture.

The Business Intelligence Investment: The Analysis Process

Blogger: Marcus Collins

In my recent post I detailed the critical success factors for a successful analysis initiative namely:

• Having the right analysis process
• Having access to the right information
• Having the right context
• Having the ability to make decisions at the right time
• Having the right leadership
• Having the right team dynamics
• Having the right people

And finally:

• Choosing the right problem

In this post I’m going to explore the “right analysis process” in more detail.

Organizations operate in a high competitive and regulated environment and so efficient and repeatable business processes are integral to an organization’s well being. This applies equally to the analysis process. The analysis process should be tailored to meet the requirements of both the organizational culture and the context of the analysis. For example, a move into or out of a market segment would require a more thorough analysis that a series of experimental what-if scenarios.

The process is shown below:

Frame the problem – requires the analyst to have a clear understanding of the problem with the problem boundaries defined and it decomposed into its constituent parts.

Design the analysis – is the selection of the appropriate technique or framework. For example, to identify the most balanced decision amongst candidates the trade study would be used.

Gather the data – identifying data that is relevant to the problem and determining the source of this information and, importantly, determining what information is not available. Data quality metrics are important as they allow the analyst to evaluating the impact of the quality metrics on the analysis output.

Execute and interpret – two different skills are used here. Execute requires an analytic discipline with a focus on quantitative fact and rule-based logic. Interpretation emphasizes human judgment. Both techniques should be used in this step of the process. The output of this step will either be actionable and the process will move onto the implementation stage or not actionable and the process will move back to the start and refine and/or reframe the original problem statement.

Implement – in context of business intelligence this is the implementation of business change. A key recommendation here is that the analysis is not the act of making decisions; rather it is a single factor in the justification for a decision. The actual decision will be a combination of quantitative information, qualitative information and human judgment.

Measure – the traditional IT view of measurement focuses on efficiency and cost. Within the context of the business intelligence initiative the focus should shift to measuring business value.

Continuous learning – the analysis process is iterative and key to the success of this is culture of continuous learning. Organizations should encourage a culture of learning through both successes and failures.

In a document to be published in early June and a series of TeleBriefings on June 2 and 3, Burton Group Senior Analyst Marcus Collins will explore the analysis process and each of the other critical success factors in more detail and provide guidance on how organizations can develop a roadmap for the successful deployment of a fact-based decision making culture.

The Business Intelligence Investment: Realizing the Benefits

Blogger: Marcus Collins

IBM recently published a report that states that while 75% of organizations recognize the opportunity for analytics only about 15% have a well developed analytic capability. This mirrors the finding of a 2008 report published by Accenture.

The current economic climate and highly competitive nature of today’s business environment requires that the traditional approach of making decision, based solely on intuition and gut instinct, be replaced by one where a rigorous information and fact based analysis process guides the decision making.

Given this landscape, what can organizations do to ensure an effective business intelligence initiative? As we analyzed both the successes and failures of business intelligence projects, a number of critical success factors emerged:

• Having the right analysis process
• Having access to the right information
• Having the right context
• Having the ability to make decisions at the right time
• Having the right leadership
• Having the right team dynamics
• Having the right people

And finally:

• Choosing the right problem

None of these factors involve tool deployment – many organizations have already made the investment in the analysis tools – rather they focus on the softer process and organizational aspects.

The key recommendations are that analysis should follow a clearly defined process with a focus on the reliable delivery of business value; that the analysis initiative should be led by the business, with strong involvement from IT and that organizations should embark on a proof of concept to both refine the internal processes and as a poster child for a wider deployment.

In document to be published in early June and a series of telebriefings on June 2 and 3, Burton Group Senior Analyst Marcus Collins will explore in detail each of these critical success factors and provide guidance on how organizations can develop a roadmap for the successful deployment of a fact-based decision making culture.

Can old-school BI deliver?

Blogger: Lyn Robison

I gotta wonder whether old-school BI -- the way we have been doing BI -- can actually deliver the information that businesspeople need to make effective decisions. Here's why:

Humpty Dumpty’s data is in pieces, and all the data warehouses and business intelligence applications can't put his data back together again, because most of those pieces of data were never designed to fit together in the first place.

To be useful, data must accurately represent reality, and to represent reality, everything must be related. Much of the source data for business intelligence applications today comes from disparate, unrelated data silos. The notion that we can consistently create accurate pictures of aggregate reality using puzzle pieces from different puzzles is demonstrably overoptimistic.

Sure, BI and DW vendors create products that let us munge data from disparate silos together, but does the resulting information goulash actually reflect reality?

It seems to me that to be useful for analysis, individual silos of data need to fit within a larger information fabric. Burton Group has produced some guidance on this recently, and will produce more in the coming months. But I would like to hear your thoughts...

Powerful Business Intelligence

Blogger: Lyn Robison

In a TED presentation, Hans Rosling shows the power of analytics. This presentation is nearly three years old, but it is quite compelling.

Hans Rosling’s ideas in this presentation about data fit very well with two concepts we have developed here in DMS:

1. MODS – the Methodology for Overcoming Data Silos. At the 15-minute mark of the video, Rosling emphasizes the importance of searching and combining the data from various silos. That is what MODS is designed to enable.
2. An analysis process for BI that my colleague Marcus Collins describes in an upcoming overview. Rosling also creates some dazzling visualizations, which are possible because he asked the right questions in the right way. That is what Marcus’s analysis process for BI does.

You can search for MODS on the Burton Group web site to find our published guidance on the topic. In addition, an overview that explains MODS in more detail will be published in the next few weeks. Marcus’s upcoming overview is entitled "Realizing the Benefit from the BI Investment". Watch for the email alert to come out in a couple of weeks.

Realizing the value of the BI initiative - analysis when the business needs it

Blogger: Marcus Collins

The traditional view of business intelligence is of operational systems feeding a data warehouse through an extract, transform and load data pipeline. This view has been static for many years – you know when it’s fossilized when switching the order to translate, extract and load is put forward as a major breakthrough and a competitive advantage!

Increasingly organizations are realizing that this retrospective view of data this model supports is not sufficient to meet the demands of companies that need to function at internet speed.

It’s time to lose this one speed fits all business intelligence model and adopt a tiered approach tailored to the organizations business requirements, competitive environment and customer demands.

Fast

Fraudulent activity analysis of credit card transactions 24 hours after the transaction occurred will still leave the banking organization financially exposed. The analysis needs to occur in real time so that the transaction can be stopped and the retailer informed. This real-time analysis has historically been the preserve of complex event processing (CEP) vendors in the financial services sector. With the slowdown in the financial sector we are seeing CEP vendors looking to expand out of this niche into fraud detection, intelligence gathering, telecommunications network monitoring etc.

Medium

Organizations that need to react to events or analyze the last few hours’ worth of transactions should look at solutions that update their OLAP cubes in near real-time. When you bid on Google AdWords you will want to analyze the new few hours’ worth of clickstream data to see if you bid was successful or valuable to the overall marketing agenda. This requirement has usually been partially met by traditional ETL and data warehouse vendors but we are beginning to see a number of smaller niche vendors entering this medium throughput sector.

Slow

There is still a class of analysis where daily or monthly trending is appropriate. Long term trending of inventory levels with a strategy to reduce the working capital requirements of the business,; guiding an expansion into new products, services or investment in existing products, services; gaining a deeper understand of customer behavior, demographics etc. and tailor offerings to gain maximum revenue from all customer interactions or offering tailored products to customers competitors see as too risky and unprofitable. All these require long term trend analysis and the traditional enterprise data warehouse approach is usually appropriate.

So, when you review your BI strategy or embark on a new BI initiative pay close attention to the timeliness of the data updates to ensure that they meet the needs of the business. Timeliness is just one of the factors to consider and in future blogs we look at the other aspects of BI that needs to be considered to maximize the value and realize the promise of analytics in today’s fast moving and competitive environment.

To recap - ensure that your architecture and BI products support the businesses need for speed!

Microsoft to Acquire DATAllegro: Leaders in data warehousing team to provide large-scale business intelligence solutions.

The data-centric specialist vendor dance floor is getting sparse...

Microsoft Corp. today announced that it intends to acquire DATAllegro Inc., a provider of breakthrough data warehouse appliances. The acquisition will extend the capabilities of Microsoft’s mission-critical data platform, making it easier and more cost-effective for customers of all sizes to manage and glean insight from the ever-expanding amount of data generated by and for businesses, employees and consumers.

“DATAllegro is a tremendously innovative company that has started to redefine the data warehouse market,” said Ted Kummert, corporate vice president of the Data and Storage Platform Division at Microsoft. “Microsoft SQL Server 2008 delivers enterprise-class capabilities in business intelligence and data warehousing, and the addition of the DATAllegro team and its technology will take our data platform to the highest scale of data warehousing.”

Microsoft to Acquire DATAllegro: Leaders in data warehousing team to provide large-scale business intelligence solutions.

BBC NEWS | Technology | Government launches data mash-up

Sign of the times...

The UK government has launched a competition to find innovative ways of using the masses of data it collects.

It is hoping to find new uses for public information in the areas of criminal justice, health and education. The Power of Information Taskforce - headed by cabinet office minister Tom Watson - is offering a £20,000 prize fund for the best ideas.

To help with the task, the government is opening up gigabytes of information from a variety of sources.

BBC NEWS | Technology | Government launches data mash-up

Prototype - How a G.P.S. Can Predict Group Behavior - NYTimes.com

An interesting snapshot of "reality mining"

Just this month, the journal Nature published a paper that looked at cellphone data from 100,000 people in an unnamed European country over six months and found that most follow very predictable routines. Knowing those routines means that you can set probabilities for them, and track how they change.

“What we do is really not random, even though it may appear random,” says Albert-László Barabási, a physicist at Northeastern University who is one of the paper’s authors.

It’s hard to make sense of such data, but Sense Networks, a software analytics company in New York, earlier this month released Macrosense, a tool that aims to do just that. Macrosense applies complex statistical algorithms to sift through the growing heaps of data about location and to make predictions or recommendations on various questions — where a company should put its next store, for example. Gregory Skibiski, 34, the chief executive and a co-founder of Sense, says the company has been testing its software with a major retailer, a major financial services firm and a large hedge fund.

p.s. Albert-László Barabási's Linked: How Everything Is Connected to Everything Else and What It Means is a classic, imho

Blogger: Peter O'Kelly

Prototype - How a G.P.S. Can Predict Group Behavior - NYTimes.com