Data Management Strategies

Can old-school BI deliver?

Blogger: Lyn Robison

I gotta wonder whether old-school BI -- the way we have been doing BI -- can actually deliver the information that businesspeople need to make effective decisions. Here's why:

Humpty Dumpty’s data is in pieces, and all the data warehouses and business intelligence applications can't put his data back together again, because most of those pieces of data were never designed to fit together in the first place.

To be useful, data must accurately represent reality, and to represent reality, everything must be related. Much of the source data for business intelligence applications today comes from disparate, unrelated data silos. The notion that we can consistently create accurate pictures of aggregate reality using puzzle pieces from different puzzles is demonstrably overoptimistic.

Sure, BI and DW vendors create products that let us munge data from disparate silos together, but does the resulting information goulash actually reflect reality?

It seems to me that to be useful for analysis, individual silos of data need to fit within a larger information fabric. Burton Group has produced some guidance on this recently, and will produce more in the coming months. But I would like to hear your thoughts...

Powerful Business Intelligence

Blogger: Lyn Robison

In a TED presentation, Hans Rosling shows the power of analytics. This presentation is nearly three years old, but it is quite compelling.

Hans Rosling’s ideas in this presentation about data fit very well with two concepts we have developed here in DMS:

1. MODS – the Methodology for Overcoming Data Silos. At the 15-minute mark of the video, Rosling emphasizes the importance of searching and combining the data from various silos. That is what MODS is designed to enable.
2. An analysis process for BI that my colleague Marcus Collins describes in an upcoming overview. Rosling also creates some dazzling visualizations, which are possible because he asked the right questions in the right way. That is what Marcus’s analysis process for BI does.

You can search for MODS on the Burton Group web site to find our published guidance on the topic. In addition, an overview that explains MODS in more detail will be published in the next few weeks. Marcus’s upcoming overview is entitled "Realizing the Benefit from the BI Investment". Watch for the email alert to come out in a couple of weeks.

Realizing the value of the BI initiative - analysis when the business needs it

Blogger: Marcus Collins

The traditional view of business intelligence is of operational systems feeding a data warehouse through an extract, transform and load data pipeline. This view has been static for many years – you know when it’s fossilized when switching the order to translate, extract and load is put forward as a major breakthrough and a competitive advantage!

Increasingly organizations are realizing that this retrospective view of data this model supports is not sufficient to meet the demands of companies that need to function at internet speed.

It’s time to lose this one speed fits all business intelligence model and adopt a tiered approach tailored to the organizations business requirements, competitive environment and customer demands.

Fast

Fraudulent activity analysis of credit card transactions 24 hours after the transaction occurred will still leave the banking organization financially exposed. The analysis needs to occur in real time so that the transaction can be stopped and the retailer informed. This real-time analysis has historically been the preserve of complex event processing (CEP) vendors in the financial services sector. With the slowdown in the financial sector we are seeing CEP vendors looking to expand out of this niche into fraud detection, intelligence gathering, telecommunications network monitoring etc.

Medium

Organizations that need to react to events or analyze the last few hours’ worth of transactions should look at solutions that update their OLAP cubes in near real-time. When you bid on Google AdWords you will want to analyze the new few hours’ worth of clickstream data to see if you bid was successful or valuable to the overall marketing agenda. This requirement has usually been partially met by traditional ETL and data warehouse vendors but we are beginning to see a number of smaller niche vendors entering this medium throughput sector.

Slow

There is still a class of analysis where daily or monthly trending is appropriate. Long term trending of inventory levels with a strategy to reduce the working capital requirements of the business,; guiding an expansion into new products, services or investment in existing products, services; gaining a deeper understand of customer behavior, demographics etc. and tailor offerings to gain maximum revenue from all customer interactions or offering tailored products to customers competitors see as too risky and unprofitable. All these require long term trend analysis and the traditional enterprise data warehouse approach is usually appropriate.

So, when you review your BI strategy or embark on a new BI initiative pay close attention to the timeliness of the data updates to ensure that they meet the needs of the business. Timeliness is just one of the factors to consider and in future blogs we look at the other aspects of BI that needs to be considered to maximize the value and realize the promise of analytics in today’s fast moving and competitive environment.

To recap - ensure that your architecture and BI products support the businesses need for speed!

Microsoft to Acquire DATAllegro: Leaders in data warehousing team to provide large-scale business intelligence solutions.

The data-centric specialist vendor dance floor is getting sparse...

Microsoft Corp. today announced that it intends to acquire DATAllegro Inc., a provider of breakthrough data warehouse appliances. The acquisition will extend the capabilities of Microsoft’s mission-critical data platform, making it easier and more cost-effective for customers of all sizes to manage and glean insight from the ever-expanding amount of data generated by and for businesses, employees and consumers.

“DATAllegro is a tremendously innovative company that has started to redefine the data warehouse market,” said Ted Kummert, corporate vice president of the Data and Storage Platform Division at Microsoft. “Microsoft SQL Server 2008 delivers enterprise-class capabilities in business intelligence and data warehousing, and the addition of the DATAllegro team and its technology will take our data platform to the highest scale of data warehousing.”

Microsoft to Acquire DATAllegro: Leaders in data warehousing team to provide large-scale business intelligence solutions.

BBC NEWS | Technology | Government launches data mash-up

Sign of the times...

The UK government has launched a competition to find innovative ways of using the masses of data it collects.

It is hoping to find new uses for public information in the areas of criminal justice, health and education. The Power of Information Taskforce - headed by cabinet office minister Tom Watson - is offering a £20,000 prize fund for the best ideas.

To help with the task, the government is opening up gigabytes of information from a variety of sources.

BBC NEWS | Technology | Government launches data mash-up

Prototype - How a G.P.S. Can Predict Group Behavior - NYTimes.com

An interesting snapshot of "reality mining"

Just this month, the journal Nature published a paper that looked at cellphone data from 100,000 people in an unnamed European country over six months and found that most follow very predictable routines. Knowing those routines means that you can set probabilities for them, and track how they change.

“What we do is really not random, even though it may appear random,” says Albert-László Barabási, a physicist at Northeastern University who is one of the paper’s authors.

It’s hard to make sense of such data, but Sense Networks, a software analytics company in New York, earlier this month released Macrosense, a tool that aims to do just that. Macrosense applies complex statistical algorithms to sift through the growing heaps of data about location and to make predictions or recommendations on various questions — where a company should put its next store, for example. Gregory Skibiski, 34, the chief executive and a co-founder of Sense, says the company has been testing its software with a major retailer, a major financial services firm and a large hedge fund.

p.s. Albert-László Barabási's Linked: How Everything Is Connected to Everything Else and What It Means is a classic, imho

Blogger: Peter O'Kelly

Prototype - How a G.P.S. Can Predict Group Behavior - NYTimes.com

Size matters: Yahoo claims 2-petabyte database is world's biggest, busiest

Remember when a gigabyte seemed big?...  See this page for the full article.

Take Yahoo Inc.'s 2-petabyte, specially built data warehouse, which it uses to analyze the behavior of its half-billion Web visitors per month. The Sunnyvale, Calif.-based company makes a strong claim that it is not only the world's single-largest database, but also the busiest.

Based on a heavily modified PostgreSQL engine, the year-old database processes 24 billion events a day, according to Waqar Hasan, vice president of engineering in Yahoo's data group.

Size matters: Yahoo claims 2-petabyte database is world's biggest, busiest

Crunching Data in a Hard-Hitting Sport - New York Times

Gridiron data mining -- see the full article for more details

They designed the software to produce what they described as the “game-winning chance.” It can also be tailored to analyze player acquisitions based on a team’s salary cap information and individual needs. Would a team, for instance, be more likely to win more games with a new defensive tackle or a new offensive guard?

Zeus analyzes a decision by simulating the outcome over and over. If it finds that a run play produces a two-game-greater winning chance than a punt, it means that if the same situation were confronted 100 times, the run play would produce two more victories.

Crunching Data in a Hard-Hitting Sport - New York Times

Guessing the Online Customer’s Next Want - New York Times

A timely NYT snapshot of the mainstreaming of BI technologies; see the full article for more details

The search for a better recommendation continues with numerous companies selling algorithms that promise a retailer more of an edge. For instance, Barneys New York, the upscale clothing store chain, says it got at least a 10 percent increase in online revenue by using data mining software that finds links between certain online behavior and a greater propensity to buy.

Using a system developed by Proclivity Systems, Barneys used data about where and when a customer visited its site and other demographic information to determine on whom it should focus its e-mail messages.

Blogger: Peter O'Kelly

Guessing the Online Customer’s Next Want - New York Times