I’m sitting here in Boston listening to all the weather prognosticators about Hurricane Earl which is currently bearing down on the Eastern seaboard. What’s fascinating is the wide variance of reports; I swear some of these reporters are being paid by Home Depot so people buy more “storm supplies.” While a hurricane is no laughing matter, it gets me thinking about the data “hurricane” sweeping across companies worldwide.
Hurricanes are like big data problems; no matter how much the forecasters say, many people tend to wait until the last minute to do something about it. We see this all the time; systems going into production without a process for data quality, which usually shows up in UAT (user acceptance testing). The system works “fine” but the data the users are getting is low quality. Why is this so often ignored? Mostly because the IT organization may not know what constitutes “high quality” data.
Take this example. We closed a deal yesterday, but the customer sent us a purchase order to our old address, which we moved out of two years ago. In the past two years, we’ve had numerous maintenance, support, services and training bills go to this customer, all of which have been remitted to the proper address. Why didn’t this particular system have the right address? Because this customer has no “tops down” data governance in place. To the IT department, the data is perfectly valid; but because there is limited or no sharing across systems, the data is actually “poor” quality data.
So first thing you say is well, they’re your customer, aren’t they using your products? They are, but not with their financials. This particular problem only shows up at the user level, not at the IT level. We provided them feedback that the address needs to be updated, but I’ll bet it happens again.
The bad data problem only shows up when it has a big impact, much like a Hurricane — that’s when you find out the seawall is too weak, that you should have put away the lawn furniture, etc. With the explosive growth of data being generated, many IT departments are facing a constant hurricane of bad data.