In 2012, we surveyed attendees at TDWI World Conferences regarding topics related to data warehousing: how they handle change, how long it takes to deploy new data, costs associated with supporting the data warehouse and so on. Over this period we received well over 500 responses and have blogged about it and shared information in other places such as infographics on our website.
The 2013 Teradata Partners Conference was billed as the largest data warehousing conference in the world. Since it had been about one year since we collected data I thought it would be worth a look-see to find out if the results were different.
I’m sorry to report that not much has changed.
One of the questions we asked was how long it would take to integrate a new source of data into the warehouse. The responses from the TDWI set were similar to what the Teradata customers said: 26.4% vs. 37.5% in the survey of TDWI attendees said they could do it in 1-4 weeks, which is interesting that the Teradata response is actually better than the TDWI responses. 34.4% at Teradata Partners and 35.5% at TDWI said between 1-3 months. 27.3% of TDWI and 25.0% of Teradata responders said it took over 3 months to integrate a new source. 10.7% of TDWI responses said they could integrate data in less than a week, but only 3.1% of Teradata customers can. The results are generally similar, so I interpret this as not much having changed. There has certainly not been a big shift towards it getting faster to respond.
But the more stunning result was the difference in the area of cost and resources to maintain the data warehouse.
55.6% of the Teradata respondents said they spent over $1 million annually to maintain their data warehouse. This is more than twice the response rate we saw from our TDWI respondents, only 25.9% of whom spend over $1 million. However, 54.6% of the Teradata respondents said they require more than 20 resources to support it, whereas only 32.7% of TDWI respondents required that many. The number of resources to support the warehouse is a definite driver of cost, but something else is driving it higher with Teradata. It might be that Teradata warehouses tend to be larger than average and therefore cost more, or it might be an indication of relative complexity. Whatever the reason, these results aren’t particularly promising for data warehousing in general. We are still spending too much money and putting too many resources into building and maintaining the data warehouse than we need to.
The emergence of a class of tools – including Kalido – that automate many data warehouse deployment and maintenance processes offer a promise of reducing the time, effort and resources required. As an example, Kalido has a Teradata customer who deployed 4 subject areas in only 9 months – quite impressive when you consider it can take twice as long for a single subject area using a traditional approach.
Perhaps the winds of change are starting to pick up. For their part, Teradata recognize the value of this automation approach, and last year entered into a global reseller agreement with Kalido. The growth of a data warehouse automation tools market would be good for customers and good for the industry.