Comparing Swim & Apache Flink
by Simon Crosby, on May 7, 2021 9:00:00 AM
Like (Apache 2.0 licensed) Swim, Apache Flink is a distributed processing engine for stateful computation on unbounded and bounded data streams. Both run distributed applications using clustered resources, and perform computation in-memory. Both can operate at large scale. Swim differs in a couple of key dimensions: Swim applications are active graphs, with deep contextual insight, enabling them to easily spot correlations in space or time, and accurately learn and predict based on context.
The largest Swim implementation that I’m aware of analyzes and learns from 4PB of streaming data per day, with insights delivered within 10ms. I’ve seen Flink enthusiasts tout big numbers too.
The types of applications that can be built with and executed by any stream processing platform are limited by how well the platform manages streams, computational state, and time. We also need to consider the computational richness the platform supports - the kinds of transformations that you can do - and the contextual awareness during analysis, which dramatically affects performance for some kinds of computation. Finally, since we are essentially setting up a data processing pipeline, we need to understand how easy it is to create and run an application.
With regards to streaming data, both Swim and Flink support analysis of
- Bounded and unbounded streams: Streams can be fixed-sized or endless.
- Real-time and recorded streams: There are two ways to process data – “on-the-fly” or “store then analyze”. Both platforms can deal with both scenarios. Swim can replay stored streams, but we recognize that doing so affects meaning (what does the event time mean for a recorded stream?) and accessing storage (to acquire the stream to replay) is always desperately slow.
Swim does not store data before analysis – computation is done in-memory as soon as data is received. Swim also has no view on the storage of raw data after analysis: By default Swim discards raw data once it has been analyzed and transformed into an in-memory stateful representation. But you can keep it if you need to - after analysis, learning and prediction. Swim supports a stream per source of data - this might be a topic in the broker world - and billions of streams are quite manageable. Swim does not need a broker but can happily consume events from a broker, whereas Flink does not support this.
Every useful streaming application is stateful. (Only applications that apply simple transformations to events do not require state - the “streaming transfer and load” category for example.) Every application that runs business logic needs to remember intermediate results to access them in later computations. The key difference between Swim and Flink relates to what state is available to any computation.
Here’s where Swim stands head and shoulders above every other approach: In Flink the context in which each event and previous state retained between events is interpreted is related to the event (and its type) only. An event is interpreted using a stateful function (and the output is a transformation of the sequence of events). A good example would be a counter or even computing an average value over a series of events. Each new event triggers computation that relies only on the results of computation on previous events.
Swim recognizes that real-world relatedness of things such as containment, proximity and adjacency are key: Joint state changes are critical for deep insights, not just individual changes of independent “things”. Moreover, real-world relationships between data sources are fluid, and based on continuously changing relationships the application should respond differently. The dynamic nature of relationships suggests a graph structure to track relationships. But the graph itself needs to be fluid and computation must occur “in the graph”.
In Swim each event is processed by a stateful actor (a Web Agent) specific to the event source (a smart “digital twin”) that is itself linked to other Agents as a vertex in a fluid in-memory graph of relationships between data sources. These links between Web Agents represent relationships, and are continuously and dynamically created and broken by the application based on context. A Web Agent can compute at any time on new data, previous state, and the states of all other agents to which it is linked. So an application is a graph of Web Agents that continuously compute using their own states and the states of Agents to which they are linked. Examples of links may help: Containment, geospatial proximity, computed correlations in time and space, projected or predicted correlations and so on - these allow stateful computation on rich contextual information that dramatically improves the utility of solutions, and massively improves their accuracy.
A distributed Swim application automatically builds a graph directly from streaming data: Each leaf is a concurrent actor (called a Web Agent) that continuously and statefully analyzes data from a single source. Non-leaf vertices are concurrent Web Agents that continuously analyze the states of data sources to which they are linked. Links represent dynamic, complex relationships that are found between data sources (eg: “correlated”, “near” or “predicted to be”).
A Swim application is a distributed, in-memory graph in which each vertex concurrently and statefully evolves as events flow and actor states change. The graph of linked Web Agents is created, directly from streaming data and their analysis. Distributed analysis is made possible by the WARP cache coherency protocol that delivers strong consistency without the complexity and delay inherent in database protocols. Continuous intelligence applications built on Swim analyze, learn, and predict continuously while remaining coherent with the real-world, benefiting from a million-fold speedup for event processing while supporting continuously updated, distributed application views.
Finally, event streams have inherent time semantics because each event is produced at a specific time. Many stream computations are based on time, including windows, aggregations, sessions, pattern detection, and time-based joins. An important aspect of stream processing is how an application measures time, i.e., the difference of event-time and processing-time. Both Flink and Swim provide a rich set of time-related features, including event-time and wall-clock-time processing. Swim has a strong focus on real-time applications and application coherence.