Survey Results on ETL-Based Data Warehouses

During 2012 we have been surveying TDWI World Conference attendees about their data warehousing investments. We wanted to know their thoughts and experiences on a variety of topics, ranging from whether they felt like IT and “the business” are on the same page in terms of keeping the warehouse up to date with the latest requirements, how long it takes to keep the warehouse current, and how many resources they expended on supporting their environment. We’ve received a little over 400 responses from events in Las Vegas, Chicago, San Diego and Boston.

You can see some of the statistics in our infographic.

The results indicate that there is still a disconnect between IT and their business counterparts in terms of their alignment. We think that a lot of this disconnect has to do with absence of a common language between the two sides who normally speak different “languages” when describing requirements for their BI and data warehouse systems. Readers of my prior posts, Business models drive agility and How to build a business model, will know that we suggest a business model as a tool to use to bridge this gap.

Another surprising statistic we found was that nearly one third of companies employ over 20 people to maintain their BI environment. We asked the survey respondents to include the number of ETL developers, BI developers and others related to the data warehouse. Furthermore, this segment was easily the largest number of responses. Only one-quarter said they employed from 6 to 10 people.

When it came down to responsiveness, two-thirds take more than one month to deliver new views after a re-organization, and nearly the same number – 64% – need more than one month to integrate a new data source into their warehouse. Given the large number of ETL jobs likely in place to build and maintain the warehouse, it probably should not be surprising that these activities take significant amounts of time and effort. But if you have more than 20 people on the team and you still take a month or more to keep your BI reports up to date, then clearly you have a complex environment and likely not very much automation to assist.

We also asked about annual costs to maintain the data warehouse. As you can see, it is not a small number. We asked respondents to consider outside consultants, software license and maintenance fees in this number, but not internal employee costs.

My conclusion from looking at the survey responses is there is still a long way to go for companies to achieve greater agility from their data warehouse. Even though just over half agree that business and IT are aligned, most are not able to deliver very quickly when there is a required change, either to deliver a new view or to bring in a new data source. This would imply that the tools they are using for the data warehouse are getting in the way. The underlying database platform may be part of the issue – and we are seeing more interest in high performance platforms such as Teradata and Oracle Exadata – but switching to a faster platform doesn’t address the fundamental issue of delivering a new view or integrating a new data source faster. The results of our survey imply that many companies are bound by their manual, brittle approach to integrating data using ETL tools.

We’re going to continue to run this survey at future events, and watch how the results change over time. So next time you see us at a TDWI event, stop by our booth and ask to take our survey. We’ll be sure to update the results with our new findings on kalido.com.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *