Does the BI Industry have it all wrong?

Big BI (defined by the small group of large but influential BI technology companies that we all know) has painted itself into a corner that is very difficult to escape from. As a result they are slowing down innovation and, more importantly, the processes used to implement true business intelligence. The mature and established BI players have built their fortunes on developing and selling large data-centric tools, applications, and services to the enterprise. In lockstep with Big BI the professional service providers and resulting industry experts have all developed methodologies and approaches they espouse and use to implement these technologies. Both the BI companies and the solution providers have a vested economic interest in continuing to see these technologies and approaches purchased and implemented. So what happens when we start to question how effective these technologies and approaches truly are to actually helping the business? And, even worse, what if we conclude that they are no longer (or never were) optimal solutions to truly address the business problems we were trying to solve?

Let me illustrate this problem with a concrete example that has become painfully obvious with the popularity of dashboard technologies. Dashboards are probably the first true end-user tool that can effectively deliver business intelligence data to every user within the organization with very little friction and low to no training costs. Done correctly, dashboards can be very effective business intelligence tools. Done incorrectly they can be little more than superficial window dressing. But the epiphany I came to a few years ago is that you can build a highly effective and maintainable dashboard solution with a very simple BI infrastructure. You do not need a data warehouse, you do not need an OLAP cube or a sophisticated ETL process. This is not to say that these technologies are not immensely helpful, and in some cases even necessary, but in almost every case they are the exact wrong place to start for a company or department pursuing its first BI initiative. I can almost guarantee you will never hear this from any person trying to sell you a BI solution. Why is that?

If the bread-and-butter of your revenue stream and expertise is selling large and complex BI systems and services that focus on all the inherently complex issues with the integration and dissemination of data, do you start telling your prospects that all of that stuff isn't really the right place to start, especially if your product line blindly assumes these pieces are necessary? This is exactly what is happening today, although I do think it is starting to change.

If a business tries to purchase any of the top-tier dashboard solutions from any of the major BI vendors they will be told that they must first implement the rest of their "stack" which will consist of myriad back-end data-centric tools that all or even a fraction of the functionality in the rest of the back-end stack may not be needed, they cannot tell you that. If they did, how would they explain this to all of their current customers who were told these significant investments were required to produce an effective dashboard? What would the impact be to their revenue stream when they suddenly told prospects that they no longer need their most expensive and profitable products and instead could purchase what was being sold as the sizzle as opposed to the steak? This same conundrum holds true for most of the industry experts who have built their careers on developing and evangelizing rigorous disciplines and techniques to build these data Taj Mahals. What do they do now? Say "Oops, sorry we were wrong, all that stuff we told you to do for the past 15 years is really not the most effective approach -- we got it backwards?"

Are Dashboards pointing the way to the future of BI?

In realizing that we really need to start top down and make the focus about the user and not the data, what has become evident to me is that dashboards, more than anything else in BI, are the most effective tools for validating your user's needs. Unlike written specifications, data diagrams, and other typical requirement artifacts dashboards provide users something they can touch and play with and let you know immediately if you are on or off track in meeting their needs. When executed properly, dashboards become the meeting ground between company strategy and company execution. Because (good) dashboards are designed to be intuitive and highly focused they force a lot of critical issues to the surface and require the business to ask questions like "Why is this important to us, what problems does it help us solve?" Additionally, on the back end, dashboards help to clearly answer the engineering questions of what data is really needed to support users, where the data is to come from and at what frequency, as well as the most appropriate abstract structures to organize data by.

For these reasons I strongly advocate that clients embarking on any business intelligence project either large or small start with their dashboard and/or report design first. Before worrying about what data exists and what format it is in (and even worse having your design influenced by what data does NOT exist) define your project in terms of the user interface that will be used to meet your business requirements for the people actually using the solution, as opposed to the engineers building it. With smaller sets of relevant sample data and user requirements focused on solving the true needs of the business community you can use your initial dashboard prototypes as a tool to not only provide some immediate user utility, but also as a set of very specific business and technology requirements for larger back-end BI initiatives. One of the reasons that this technique is so effective, is that it provides the ability to iterate and test assumptions with the user community with significantly less cost and time than it takes using a bottom up approach. It takes days or, worst case, weeks to prototype a dashboard, while it takes months or even years to create an integrated data warehouse.