If there is one question we hear from almost every Dataspace client, it is, “Do I have big data?” Many clients are torn about the prospect of big data. On the one hand, having big data represents a sort of badge of honor, i.e. “We do so much business and capture so much data that we have BIG data!” On the other hand, having big data is like a rash – you don’t really want it, but you’re stuck with it. So, I guess that big data is sort of a rash of honor.
What Is Big Data?
Years ago, Doug Laney, at The Meta Group (now Gartner), wrote what has morphed into the accepted definition of big data…
Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
(By the way, if my memory serves me, I believe the original definition used “high-volume, high-velocity and/or high-variety” rather than just “and”.)
Doug’s definition is really on target because it not only defines the problem / opportunity but also tells you what to do. It says that if your data has any of the three v’s, volume – velocity and/or variety – you need to consider “innovative forms of information processing”.
What This Means
If you think about it, this big data concept is not fixed, but relative. Big data to one organization may not be big data to another. If the technologies you use today allow you to do what you need to do with it, then either you don’t have big data, or you have already implemented the appropriate technologies to allow your organization to fully leverage that data for decision-making. If your organization needs to put your data to work in ways that are not possible today, then you have big data that is currently underutilized, most likely because of technology deficiencies.
What the definition gives you is permission to say, “Hey, to do what we need to do, I have to look at some new technologies.” And, it gives you the argument to take to management to justify the related investment.
If you get wrapped up in the hype of big data you could miss the opportunity. In other words, don’t waste your time worrying about whether you have big data and specing out big data vendors. Instead, spend your time figuring out what the business really needs and finding the best technology to address those needs. Whether it’s marketed as a big data technology or not, ultimately the right solution will allow your organization to leverage your data assets for better decision making.
My Recommendation
In the end, it really doesn’t matter if you have big data or not. What you have is data and you should capitalize on it. If capitalizing means investing in new technologies, and the ROI is there, do it! However, first make sure that your current technology stack won’t get you where you need to be without expanding on it. This is especially important as our legacy vendors continue to bake cool big data technologies, like in-memory and column store data management, into their core offerings.