From Sensors To Cloud: Making The Data Flow Smooth

As modern enterprises plan for Robotics and other IoT tools, they are forced to rethink their platforms to ensure that data flows smoothly from devices to edge to cloud. The need to remove bottlenecks anywhere in the path of data to satisfy modern day data needs, the result of today’s technologies reshaping customer expectations for real-time results and increasing reliance on analytics driven automation. 

Using sensor data is not something new but the ground underneath has shifted with a focus on the 5 Vs of industrial internet era:

• (Data) Volume
• (Data) Velocity
• (Data) Variety
• (Data) Veracity
• (Immediate) Value

Modern data driven business requires a rethink in the underlying plumbing to support the 5Vs mentioned above.

Embedded SwimOS for modern data architectures

As enterprises use IoT devices in their organization needing real-time analytics, the underlying plumbing should be reimagined along with the architecture. There are arguments made about how REST is too limited for a world of real-time streaming data. Swim’s DataFabric uses the WARP protocol to overcome these limitations and replace the REST based data ingestion interface. SwimOS and the DataFabric brings together APIs from all data sources and data destinations into a stateful service mesh like architecture with WARP enabling high performance analytics by reducing the latency for large volumes of data (large volume and high velocity).

In the industrial internet era, we will see environments with thousands of IoT devices interacting with Edge and/or Cloud. These environments are dynamic with large volumes of data moving fast from the IoT devices to Edge and to cloud. Unless the data flow is smooth, the output will be suboptimal, even leading to disruption and financial loss. Even today, organizations are relying on platforms that use REST interface to aggregate data, analyze and deliver the customer experiences. As we pointed out in the 5 Vs of industrial internet, immediate Value is one of the critical requirements. This requires an end to end architecture that enables real-time streaming that can withstand the needs of fast and high volume data. 

The traditional approach to using cloud as data storage and analytics engine through their REST interface or using platforms that use Apache Tomcat to serve the requests for data will not work. These data ingestion interfaces choke at high volume and speed. We recommend SwimOS and the DataFabric as the alternate solution handling the data ingestion needs of modern architectures. Swim DataFabric creates a stateful mesh of streaming APIs from source to destination making the data flow smoothly, providing the immediate Value needed in modern industrial environments.

Let us consider two scenarios where SwimOS can be embedded and seamlessly handle the data flow between sensors and other endpoints.

• An environment with tons of sensors and IoT devices sends data to a gateway at the edge where the data is processed and then stored in the cloud. By embedding SwimOS on the IoT devices, Edge gateway and the cloud, a streaming service mesh of APIs is created where data can flow seamlessly with physics being the only bottleneck. Such an architecture not only reduces latency and increases performance, it also offers more control for optimization. Such a mesh network not only optimizes the data path but also makes it easy to set up analytics based automated actions
• Another scenario could be an environment where IoT sensors send their data directly to the cloud for processing. For example, an organization may use Azure IoT to process the data coming from an assembly line. By embedding SwimOS on the IoT devices on-premise and in the cloud, a hybrid mesh of streaming APIs is created that reduces the friction in the data flow for near real-time processing of the data

Using Swim DataFabric offers the following advantage in the above mentioned architectures and multiple other scenarios where data flows in a network for real-time processing:

• No bottlenecks in data path with very low latency
• Higher performance
• Ability to scale, handling data from millions of devices
• Lower costs
• Much better control over data path for optimization
• Real-time automation

The modern industrial internet requires organizations to change the architecture of data infrastructure to allow for a more dynamic system and data flow across networks. Such systems will fail or perform in a suboptimal way with a REST based data ingestion interface (eg: Apache Tomcat serving data ingestion). SwimOS and DataFabric both provide a stateful runtime that’s optimized for high volumes of streaming data and is well suited to handle such use cases.

Learn More

Disclosure: This was originally posted to Stacksense.io. Swim.ai is a client of Rishidot Research.