How Streaming Data Changes Our Assumptions About Application Architecture

by Brad Johnson, on Jan 7, 2020 10:51:20 AM


It’s 2020 and we’re still beholden to application patterns established in the era of mainframe computing. The Web was originally created to exchange static resources between user agents. The Hypertext Transfer Protocol (HTTP) and Representational State Transfer (REST) standardized the way static resources are exchanged between user agents. 

But today’s applications are no longer primarily concerned with exchanging static resources. Today, the primary task of applications is synchronizing state between user agents. However, there currently is no analog to REST and HTTP for synchronizing the real-time state of user agents.

Why Streaming Changes Everything

The Web is always evolving to support new purposes. We’ve evolved from documents to web pages, and from web pages to interconnected social networks. Today, we’re in the midst of another transformational change: evolving from static data to streaming data. And this shift to streaming data requires us to rethink the way we’ve been building applications.

Real-time batch architectures can struggle to keep up with message rates. This can lead to unmanageable queues and buffer bloat. But streaming data is continuous, and can compound these inefficiencies, causing performance drop offs and skyrocketing application latencies. Furthermore, applications continue to grow increasingly decentralized and complex. 

In order to combat these challenges, we need a new architecture optimized for continuous data flow across distributed compute. This new architecture would optimize for high volumes of state changes, and present a real-time model of these changes for application developers to program against. But in order to do so, we need to rethink the standard models for application design.

In other words, our old assumptions no longer apply:

Old Assumptions

New Assumptions

  • Resources change infrequently
  • State changes continuously
  • Content is primarily human generated
  • Content is primarily machine generated
  • Memory is expensive
  • Memory is cheap
  • Compute is centralized
  • Compute is decentralized
  • Networks are sparse
  • Networks are ubiquitous
  • Store everything
  • Store what is relevant
  • Act on data later
  • Act on data now
  • Storage-centric
  • Compute-centric
  • Web driven by humans
  • Web driven by reality

Learn More

Over the course of 2020, we'll be publishing an enormous collection of new swimOS tutorials, tools, documentation, examples with an eye on improving ease of use for swimOS developers. With this in mind, keep an eye on the swimOS blog for the latest product updates and news.

You can get started with swimOS here and make sure to STAR us on GitHub. You can also learn more about our enterprise product DataFabric here.

Topics:Stateful Applicationsdistributed computingHTTPweb applicationsswimOSRESTstreamingstreaming apiopen sourceopen source software