This is a part 1 of a two part series.
While the data lakes landscape has matured over the last few years, the streaming analytics market is continuing to develop as more enterprises embrace the use of streaming analytics to provide real-time insights for their business systems. Early returns suggest that embracing streaming analytics in industry can lead to significant efficiency gains and decreased operational costs. For example, a study by McKinsey & Co. found that “using real-time data to predict and prevent breakdowns can reduce [manufacturing] downtime by 50 percent.” With real-world cost savings at stake, the momentum toward real-time has never been stronger.
Forrester Research recently published their “The Forrester Wave: Streaming Analytics, Q3 2017” report, which compares streaming analytics vendors and draws conclusions about the state of the real-time data industry. In part 1 of this series, the Swim team highlights some of the key takeaways from this report, and add our own perspectives to help guide you through a rapidly evolving landscape. We will discuss how to evaluate streaming analytics solutions in part 2 of this series.
What is Streaming Analytics?
Forrester defines streaming analytics as “software that provides [the ability for] analytical operators to orchestrate data flow, calculate analytics, and detect patterns on event data from multiple, disparate live data sources [and] to allow developers to build applications that sense, think, and act in real time.” In other words, streaming analytics simply refers to the ability to process and react to streaming data from a multitude of sources, all in real-time. Forrester explains that “streaming analytics solutions identify perishable insights — insights that enterprises must act on within a given time frame or else the opportunity to change the course of business outcomes will pass. Streaming analytics solutions don't stop there though. They also offer continuous integration, orchestration of business processes, and embedded execution of code — all driven by streaming data from a multitude of data sources.”
It’s important to distinguish streaming analytics from “near real-time” technologies. “Near real-time” solutions are often database technologies which are optimized for speed, but still incur network and processing latency, delivering responses in seconds to minutes. Streaming analytics solutions, however, introduce minimal processing latency (milliseconds or less) and often exist on the edge, thus avoiding network latency as well. These streaming analytics solutions are ideal for industrial or Internet of Things (IOT) contexts. As Forrester points out, “streaming analytics solutions are particularly well suited to Internet of Things applications because they are by nature real-time and spew sensor data that can be analyzed in real time.” If an application involves multiple sensors which are continuously creating high volumes of data, then a “near real-time” solution will most likely be insufficient.
Streaming Analytics at the Edge
An important aspect of streaming analytics solution is the proximity of the streaming analytics to where the data is generated. Solutions which are capable of processing streaming data on or near physical sensors and devices are said to operate at the edge. By processing sensor data at the edge, streaming analytics solutions can process and act on sensor data without incurring latency from the network. Fast edge data streams can power cloud-based applications, or be shared with other edge nodes to inform their real-time decision-making.
Keeping latency to a minimum is important for time-critical use cases, such as when a streaming analytics solution must “detect urgent situations to act upon automatically or feed real-time dashboards for human decision makers.” For industries such as Healthcare, Defense, and Emergency Response or hazardous industries like Oil & Gas and Chemical Manufacturing, getting a timely response from a streaming analytics system can potentially mean the difference between life and death. In these cases, proximity to the edge can provide invaluable extra time to react.
This is a part 1 of a 2 part series. Read more about the future of streaming analytics in part 2.
Learn how Swim ESP™ powers Machine Learning and intelligent decision-making at the edge with a streaming analytics engine for industrial and other real-time applications.
Gualtieri, Mike, et al. “The Forrester Wave: streaming analytics, Q3 2017.” Reprints.forrester.com, Forrester Research, Inc., 7 Sept. 2017, reprints.forrester.com/#/assets/2/308/'RES136545'/reports.