A Stateful Model For Realtime Applications
by Brad Johnson, on Mar 11, 2016 3:01:00 PM
Announcing the SWIM.IT In-Motion App Fabric, an application platform for building and distributing realtime services.
In 2005, Twitter didn’t exist. Netflix was sending DVD’s in the mail, the Internet reached more than 1 billion people and Facebook only had 5.5 million active users. Contrast with today, where the publicly traded $FB has 1.6 billion users and Twitter has 300 million. There are websites that handle more traffic than the entire internet did less than a decade ago. According to research, “a full 90% of all the data in the world has been generated over the last two years.” And that was back in 2013, we’ve lapped the field since.As applications supplant websites for internet supremacy, companies will process billions of unique interactions a day across hundreds of millions of devices around the world. The stakes are high — sooner rather than later — organizations will need to modernize their IT infrastructure to put all this data to work and compete effectively.Traditional application architectures force a decision between scalability and system latency. Meanwhile, “in-motion” applications that require both fast data and massive scale are left wanting for an alternative. Modern, cloud-native applications demand greater performance to power realtime, responsive features. Forward-thinking organizations must embrace data-in-motion and look toward streaming-based architectures.
Application Needs Have Outpaced Databases
Databases and their supporting technologies were intended to serve a Distributed Hypermedia model and by extension, media-centric applications. Just like the name suggests, “databases” are oriented around data. They store, organize, address, and make accessible media and other static information. Backend software like application servers Node.js and Ruby on Rails, relational databases like MySQL, NoSQL databases like Cassandra or MongoDB, and messengers like Kafka (just to name a few) have drastically improved the performance of these media-centric applications. But not all applications are media-centric.Applications that need Process Control or other event-driven capabilities often must construct a solution built from these same media-centric technologies. However, Process Control applications differ from media-centric applications in their relationship with data. For media-centric applications, data is a static commodity that must be distributed throughout the application as needed. Instead, Process Control applications receive multiple continuous streams of data and must react to events as they happen.This is a much different paradigm than simply storing and shipping media. Because there is such a high rate of data inflow, there is also a high rate of state change. Managing such rapid changes with the traditional, stateless Hypermedia model is incredibly difficult. In order to overcome this challenge, developers have turned to constructing specialized architecture silos for each unique application feature and managing state independently for each. But that’s not all, they must then solve for the scalability of each silo before trying to scale the entire system itself. Accomplishing this is prohibitively labor intensive and extremely expensive.
Approaching Event-Driven Apps Differently
The SWIM.IT Fabric (SWIM for short) was purpose-built to perform in-motion Process Control. Using SWIM, our own Privacy Watch browser extension is able to continuously process website data and serve realtime notifications based on the results of the processed data. SWIM continuously handles high volumes of realtime data with millisecond latency, ideal for the complex event processing necessary to power Privacy Watch. It was designed to support modern applications and simplify how developers architect realtime systems.
One of the key differentiators of SWIM is its use of multi-directional, realtime APIs. With REST API-based applications (which are inherently stateless), state is confined to the database and updates are periodically pushed to the client. With SWIM, logic is centralized to the cloud and runs processes in realtime, continuously streaming the processed data to the client. This model realizes significant improvements in responsiveness and availability while eliminating the need to change code when scaling from small containers to large clusters. Because data is stored locally within cloud services, the system can be pervasively stateful and event-driven. Additionally, services are globally addressable which offloads system complexity to SWIM instead of burdening developers.
Finally, A Model For Process Control
SWIM reduces the immense technical challenge of working with services and gives developers everything they need to build and scale realtime, collaborative apps. The SWIM Fabric is a vertically integrated platform by design, so it avoids compatibility issues that arise when third-party components are updated independently.
A requirement for us when creating SWIM was that it be accessible to everyday app developers. SWIM supports a variety of programming languages and includes native SDKs for both web and mobile platforms. Because SWIM treats cloud and edge devices the same, client and server-side code are written similarly. The code developers write for SWIM looks the same regardless of whether it runs in the cloud or on a phone.
How Does SWIM Work?
SWIM can be broken down into three basic parts: Services, Lanes and Links.
Services are the building blocks of SWIM. They are individual, cloud-based processes that link internally within SWIM and with client applications. Unlike Web services, which can only be accessed periodically, SWIM services are actively and persistently linked. SWIM is engineered to run millions of services, distributed across massive clusters of machines.
SWIM services are linked to and controlled by designating SWIM lanes. SWIM lanes can be specialized with different behaviors and messaging semantics.
A SWIM link is a persistent subscription to the events on a lane of a particular SWIM service. SWIM efficiently multiplexes and prioritizes huge numbers of links across a small number of network connections.
Building Apps With SWIM
The best way to understand the mechanics of SWIM is to start building an application. If you want to build your own SWIM app, check out our GitHub for more detailed information. For an example, here’s how easy the server-side code is for a chat app like Slack:
Those two lines of code are all it takes to create a service for managing chat rooms. It maintains persistent lists of chat messages and broadcasts new messages as they’re posted. This code will be instantiated for every chat room that gets created.
To add a mechanism for creating new chat rooms, you only need a few more lines:
To integrate with a Swift app instead, use the SWIM Swift SDK:
Want To Use SWIM For Your In-Motion app?
SWIM lets developers focus on what they do best — write code — without getting stuck writing glue code or middleware. As one of our founders, Chris Sachs, says “we’re aiming to power the next generation of realtime apps.”
With realtime capabilities, apps can deliver rich, context-driven and focussed interactions with users. Using SWIM, developers are free to create more relevant and valuable experiences for their users. If you want to learn more about SWIM, the team behind it or how you can use SWIM for your app, you can take a look at our GitHub, check out our website or contact us.