See - at the Resolution of Your Data

by Simon Crosby, on Nov 16, 2020 9:38:44 AM

Mark Burgess, of CFEngine and Promise Theory fame, gave this fabulous concluding talk at DevOpsDays Oslo two years ago. He observed that hoping that a complex system is in a given state is a belief that results from a failure to observe the truth and deduce the (complex) consequences fast enough to be able to be sure what happened.   

He explains that the application of the Nyquist Theorem, which underlies standards for  sampling continuous variables (like audio at 44.1 kHz to reproduce a 22kHz signal) must also apply to monitoring of computers (and anything else, for that matter), and suggests that for complex systems we need to take on board its lessons.  The goal is to get from hoping to knowing.  But of course, the problem is that simply knowing whether a single component is running is not enough: Complex systems are composed of many interdependent parts – a dynamically changing graph.

SwimOS is an open source software platform for building stateful, real-time applications. SwimOS enables developers to create distributed applications using stateful, concurrent processes, called Web Agents. A SwimOS application is an automatically created, distributed graph of Web Agents – one for each data source - that each concurrently clean and process streaming data from a single source and analyze their resulting state changes on the fly.  These state changes are immediately observable to any app or user and are at most ½ RTT out of date.  Is that good enough? Web Agents offer an opportunity to achieve the Nyquist-like ambitions for visibility into complex systems that stream masses of data in the real world.  But there’s more.  Even a real-time view of the state of a single source in a complex system (eg: microservices in a cloud-hosted application, or RFID tagged parts on an aircraft assembly line) is meaningless.  It is the interrelationship between “things” that is key: If source A and source B are both behaving weirdly, their correlation in time may be crucial to discovering the real problem.

Swim Web Agents solve this problem by dynamically building a graph that links related Agents.  Each vertex in the graph is a Web Agent that concurrently analyzes both its own state and the states of agents to which it is linked. Links are made and broken dynamically based on changing real-world relationships between data sources, so the graph is continually in flux.  In the physical world many relationships can be inferred from a geospatial context, but links can also express simpler notions like containment or dependency.  Web Agents continuously analyze, learn and predict from their own states and those of Agents they are linked to. And they stream their insights / analysis / predictions in real-time to applications, storage and GUIs.

SwimOS provides a complete application runtime and bi-directional streaming APIs for analysis of boundless data in which Web Agents capture complex dependencies and interrelationships in the real world that they model.  So the insights they offer can be rich.  Web Agents can compute distributions, support queries and dynamically attach computational traits that allow data scientists to operate on live data. They can feed a Spark analytical model, or statefully learn and predict on their own. Their insights are available in real time in UIs and to other applications. Web Agents get us from hoping to knowing faster, by helping us to better understand the truth of dynamic relationships in a complex system.