iPioneers.com

Now is a good time given current market conditions to start thinking about ways to reducing the cost and time to market for Data Warehousing and Business Intelligence efforts. Data Warehousing has had the goal of driving towards one version of the truth for the enterprise, but these have been expensive endevours. There are ways to reduce the cost, effort and embrace change through the lifecycle of such projects. I have collected those practices under the umbrella of what I term as the Agile Data Warehouse and Business Intelligence Triad which is shown below:

Agility is the new buzzword in Software Development circles… Well maybe not so new considering that the Agile Manifesto was drafted in February 2001. The agile approaches to software development models have been packaged in methodologies such as XP, Scrum and Lean Software Development. The principles of agile software development manifest themselves in practices such as:

• Short time boxed iterations of 2 to 4 weeks
• Cross-functional self organizing teams
• Collocation of teams
• Customer representation on teams on an ongoing basis
• Daily stand-up meetings
• Retrospectives at the end of each iteration
• And more…

These practices have served many application development project teams well but only a limited amount of literature talks about how these practices can be applied to Data Warehouse/Business Intelligence projects. Most literature still talks about delivering solutions in iterations of 90 days. Even at a recent TDWI Conference (The Data Warehousing Institute is the industry’s leading trade organization) that I attended in February 2008 most of the sessions expounded the same message.

Why is that?

Is there something that makes Data Warehousing/Business Intelligence projects not suited to these practices? Is it that Data Warehouses are inherently monolithic entities that are not suited to being sliced into small slices of functionality that can only delivered in a waterfall approach.

After 3 years of working on building the data warehouse at the organization where I work, to my surprise as well, we were able to apply an agile approach to project to achieve the goal of reducing the project delivery time from 12 months for the first project all the way down to 4 months in our last major project deliverable. What I also learned in the process is that the software delivery methodology is not sufficient but there are other factors involved.

Agility solves the part of the problem of adapting to changes during the course of the project rather, than resisting to change as most project management classes teach you, where change has a dark cloud over it in the name slimy slithery name of “Scope Creep”. Agile helps you embrace change and stay focused, and to deliver in the shorter time frames that are demanded of many a BI/DW project manager two other factors beyond the software development methodology need to be taken into account: the People who are responsible for the developing the solution and the Participants outside the delivery team that influence the project.

I will detail each of the nodes in the Agile Data Warehouse and Business Intelligence Triad in the series of articles that will follow.

I am not under the impression that I have answers to all questions or that I have explained my experiences as well as I can so I am very interested in hearing from you to clarify and to learn more about the approaches you may have attempted to reduce time and effort to deliver quality solutions and what the results were. I look forward to hearing from you.