If you have huge volumes of data, chances are that you can get value out of analyzing that data. But, would you derive significant value from analyzing it in real time? This is an important question because, while there are benefits to real time analytics, there are also real costs. To help you decide whether real-time analytics make sense, consider these questions.
Does data arrive to you on a constant basis?
Remember Doug Laney / Gartner’s textbook definition of ‘big data’? It tells us that big data isn’t just about the volume of data but it’s also about the variety and the velocity of that data. So, if you’ve got high velocity (e.g. machines are always humming, trucks are always driving and customers are always buying) you may see major operational improvements through real-time analytics. On the other hand, if your company operates largely in batch mode, where the exact timing of transactions is relatively unimportant, it probably doesn’t make sense to build a real time analytics infrastructure.
Are you building rockets or launching rockets?
While both are necessary to delivering payloads, the job of building rockets is far different than the job of launching them. Building a rocket takes a long time. Plans are made, and changed, based on the data that is captured and aggregated over that long time. If a change is required, it is considered and implemented over time. The decision to abort a program could take months to make.
When launching a rocket, however, events happen and decisions must be made on time horizons of seconds. The decision to abort a launch needs to be made in seconds, or less.So, when considering a real time analytics investment, consider your situation: Is your business closer to building rockets…or launching them?
Are there decisions you would make differently based on changes that occurred in the past hour?
Imagine that you run a hospital emergency room and that you schedule a certain number of doctors to be in the facility throughout the day. One day, when you’re staffed for an average of ten patients per hour, you start seeing 12, and then 15, and then 18… What do you do? You probably start calling your on call staff to come in and pick up some of the unexpected load. In other words, you have the ability to make changes based on things you’re seeing at the present time.
Alternatively, imagine you run a railroad and you notice that you’re selling far more tickets for today’s trip to Kalamazoo than usual. You’ll likely be able to sell twice what you normally sell. However, the train you have scheduled on that route is sized to your normal capacity and you don’t have any spares. While you hate to lose the revenue, there’s likely not much you can do about it in the short term. You can’t buy a new train with a moment’s notice. Thus, in this case, knowing in real time might be somehow comforting, but it doesn’t pay much in the way of dividends.
Notice that in both cases, however, analyzing trends over a long time period might provide insights on how to better set your capacity in the future. Which raises an interesting side point: even if you benefit from real time analytics, you can also probably benefit from analyzing your data in the traditional way, too.
Do your customers or employees need data immediately after it comes available?
Have you ever entered data into a web site only to find that there’s a delay of a few hours before it becomes available for you to view. Some hotel and car reservation sites work this way. If you check for your reservation and don’t see it, you’re left wondering if it really got entered. Situations like this present a good argument for real-time analytics.
Situations like this require a good look at the dividing line between operational systems (those that do day to day transaction processing) and informational systems (those used for analysis). While the business need exists, should the data come from your operational or informational systems?
It’s sometimes helpful to consider what kind of data the user wants right after it’s entered. If it’s just atomic details (e.g. show me the reservation I just made), you may want to simply direct those queries to the original, operational system, or a real-time replica of that system. However, if the need is for data that aggregates the most recent with older data (e.g. show me the total of all my purchases for the year, including this most recent one), you may have a strong argument for real-time analytics and real-time analytic data storage.
Designing and implementing effective real-time analytical systems requires large upfront investment and significant monitoring costs. In-memory databases, scalable platforms and other technologies required for analyzing high-transaction throughput activities are not easy on the budget and so it is important to understand which areas of an organization will benefit sufficiently from real-time insights.
While it is highly probable that real-time analysis can enhance your business, not every operational activity will require it. It may be that the speed of customer interactions and business transactions flowing in and out of your operation isn’t fast enough to warrant the cost of these capabilities. A lower batch-processing frequency such as daily or biweekly processing may give you all of the actionable data you are looking for. In most companies, the customer-facing departments are those that benefit most from real-time processing, while backend activities such as purchasing and accounting are not time-sensitive enough to warrant the investment.