Exploring the Challenges of Building Real-Time Applications
by Brad Johnson, on Aug 15, 2018 5:34:09 PM
Earlier this year, Mickaël Rémond, the founder of ProcessOne, a real-time solutions provider, published a blog post titled The Challenges of Building Real-Time Applications. I highly recommend giving it a read, as it’s an excellent summation for why “many developers and software architects struggle to introduce real time in their applications or information systems.” In it, Rémond relates that “far too often, I have seen ‘real-time’ web applications that were just doing refresh queries every 5 seconds to keep the data up to date. This is a naive way to build real-time applications, but the real problem is that most of the time, it does not scale.” Real-time applications require an interplay between several complex database and networking technologies, therefore Rémond concludes that “a holistic approach [is necessary] for real-time software design. Real-time is much more than building chat systems. It needs to be placed at the heart of your application.”
The typical real-time application is a Rube Goldberg machine of different open source components.
For his own solutions, Rémond describes using a “mix [of] XMPP, MQTT, Kafka, among other protocols, to connect people, things and applications together.” This is a sensible approach, making use of tools which are familiar to most developers of real-time applications. Furthermore, Rémond is correct that a holistic approach must be employed to develop an effective design for real-time application. However, there remains an even larger challenge. Today, most real-time applications are comprised of independently optimized database components, each designed to perform a specific function. As such, these individual components must be integrated with each other, into a Rube Goldberg machine of different open source pieces. Each integration introduces bottlenecks and other inefficiencies, such as timers, buffers, and queues. Additional network hops and queuing delays are incurred, adding unnecessary (and undesired) latency at each turn. The cumulative effect of all these integrations are applications which tend to be inflexible and difficult to scale. No matter how efficiently a real-time application is designed, the very nature of building a complex system of loosely coupled open source components creates a situation counter to the goals of a real-time application.
For this reason, SWIM’s platform is vertically integrated. We take inspiration from many of the same technologies mentioned above, but SWIM has implemented an agent-based model which uniformly incorporates the various functions of traditional database architectures into composable agents which can be distributed (and redistributed) dynamically. Instead of utilizing available open source components, SWIM built a bespoke platform which optimizes for efficient integration between application functions. This approach has led to a much more flexible, scalable way of building real-time applications. In the coming weeks and months, we’ll be describing specific ways in which our platform compares and contrasts with existing open source solutions and other familiar approaches.