In my first blog post, I questioned whether performance and capacity was really the most important thing for data warehousing, and I suggested that agility and flexibility ought to be considered a more important topic. Some recent research from Boris Evelson and a blog post from Jim Kobielus, analysts at Forrester Research, underscore the benefits of an agile approach, both in data warehousing and business intelligence tools as well as in methodology.
The discussion about agility in data warehousing is very timely indeed. Last week, the U.S. Federal Reserve Board indicated that the economic recovery could take 5 to 6 years. If true, companies will look very closely at how to lower their total cost of ownership and accelerate the time to value of any IT investments. Agility in responding to change is one way to do just this.
So how do you get there?
Jim makes some very good points in his blog post, but the one I want to discuss is in this quote:
“…it would be very useful to have an underlying DW ecosystem that has the elasticity to be rapidly refactored and redeployed into any agile topology you define.
However, agility-enabling tooling and platforms are not enough. You can’t get agile simply from implementing some fancy software product. Without development process discipline an agile DW initiative can quickly degenerate into anarchy, as everybody brings their own preferred tool into the effort, nobody talks to anybody else, no one reuses anybody else’s work, and nothing interoperates…”
I think there is no doubt that the methodology is critical, but having a tool that is “agility-enabling” is going to help a lot and is not a simply a “nice to have” element in the equation. Due to the nature of traditional data warehouse development, you can follow an agile methodology but still create a very inflexible architecture and therefore be subject to long delays in responding to business change. No amount of agile methodology is going to significantly speed up editing thousands of ETL jobs.
Build it Right
So how do you avoid lots of hand-coding? One way is to use a business-model-driven tool. There are significant benefits to using a business information model to drive building the warehouse:
First, business models enable better business & IT collaboration to better meet business needs. The business model provides a common language to help business users understand what is in the data warehouse. It isn’t the technical jargon IT uses, and it models the real-world “things” and the relationships between them that business cares about. It also makes it easy for IT to capture and understand business requirements. When requirements change – and they will – the business model can easily display the change. In Kalido’s case, the business model generates the physical data model, so you gain agility to respond quickly when the requirements change.
Second, the business model promotes consistency across applications and streamlines development by creating reusable model components. This means that as you create a data mart or warehouse for a specific application or department or region you can reuse model components such as a time dimension or a product dimension for each of the marts without having to re-create it every time. Any local variations can be handled in each mart. This saves huge amounts of time and resources by avoiding rework and thus lowers costs.
Third, the business–model-driven environment accelerates time-to-value through iterative prototyping. As you develop the business model it is fast and easy to make adjustments as you go. You can quickly deploy prototypes with data, show them to the business, and make changes once the business users see what they’re getting. And you know they always find things to change! A model-driven tool such as Kalido tracks the changes to the model so you can get time variant views of information as of a certain time, as of now, or even as of a transaction.
The cycle I just briefly described significantly complements an agile methodology. Even with an agile methodology, if you are using a data warehouse tool, or set of tools, without the flexibility to help you keep pace with business, you’ll have a semblance of agility but will be applying it to an environment that will struggle to keep up.
Where’s the proof? Take it from one of our customers. Last year, Eisai presented at Kalido Connect, our annual user conference. The takeaway:
“The sales and marketing department challenged us to build a robust, flexible business intelligence infrastructure that wouldn’t take two years to build and be obsolete by the time it was complete. Our answer to them was Kalido. For example, we were able to reflect a sales force realignment within three months, when it previously would have taken us nine to 12 months. Overall, with Kalido, we’re realizing cost savings of 1.5 million dollars every year, thereby showing our management team not just the soft value, but the hard, tangible savings as well.”
This is just one attribute of an agility-enabling data warehouse tool. In subsequent blog posts I’ll cover additional elements.
What do you think? Do your current tools help or hinder agility in your data warehouse?