What is Recon? How We Augmented XML and JSON For Streaming Data

by Brad Johnson, on Aug 21, 2019 12:48:22 PM

Ever since the first applications started moving data records, we’ve needed ways to annotate those records with formatting instructions. Many of these record notation formats are familiar to developers. For example, according to the IETF, the ubiquitous JSON “defines a small set of formatting rules for the portable representation of structured data.” In practical terms, JSON makes it possible to describe value pairs, arrays or a series of values as a human readable document. 

Similarly, the prolific XML markup language makes it possible to encode data into a format that is both human and machine readable. Without formatting instructions provided by JSON and XML, machines would lack critical context to express and analyze documents. But what happens when data cannot be expressed as a document? 

For example, streaming data cannot be adequately expressed as a document. Documents are historical artifacts, a snapshot of data at a defined point in time. Conversely, streaming data is an ongoing and continuously updating series of values. Representing streaming data as a document is like trying to capture all the water from a fire hose with water buckets (without losing water pressure!). Instead of trying to represent streaming data as a document, developers need a way to annotate streaming data and include relevant state context for every event. In other words, there’s a need for a language that combines object notation with attributes. This is the motivation for the Recon notation language.

What is Record Notation (Recon)?

The name Recon is shorthand for Record notation. Recon is object notation with attributes, sort of like if JSON and XML had a baby. Recon attributes facilitate uniformly disambiguating polymorphic structures. And first-class language extensions for selectors, expressions, and functions make Recon a highly expressive format for domain specific languages. Recon is part of the open source swimOS framework, which can be found here.

Recon combines the simplicity of JSON with the expressiveness of XML. As shown in the example below, Recon looks a bit like a hybrid of the two. Yet Recon is deceptively simple: the grammar for Recon is scarcely larger than the grammar for JSON. And this underlying uniformity makes Recon more expressive, and more consistent to work with, than either XML or JSON.

Here's a Recon example:

@html { @head { @title "Greetings" } @body { @h1 "Introduction" @p [I have @a(href:"https://en.wikipedia.org/wiki/Markup_language")[markup syntax] for when you need it. But I'm not a text chauvinist. I'm a structured object notation first and foremost. The numbers {1, 2, 3} are parsed as numbers, not strings. Any my attributes make it easy to define, embed, and disambiguate microformats and domain specific languages.] @p [Need a microformat for time? You'll find it falls out naturally after of using Recon. Need to build a DSL for real-time GUI widgets? Recon helps you do so cleanly and concisely, like this:] @pie { title: "Events" linkStats: @link(host: "warp://traffic.swim.services", node: "swim:meta:mesh", lane: "linkStats", type: value) @slice { value: $max(0.1, $rate($linkStats.downMessageCount)) label: @text($percent($value, $total)) legend: @text([Down ({$round($value)}/s)]) innerRadius: 10 + 7.5 * $value / $max($value) @pct outerRadius: 20 + 7.5 * $value / $max($value) @pct } @slice { value: $max(0.1, $rate($linkStats.upMessageCount)) label: @text($percent($value, $total)) legend: @text([Up ({$round($value)}/s)]) innerRadius: 10 + 7.5 * $value / $max($value) @pct outerRadius: 20 + 7.5 * $value / $max($value) @pct } } } }

Data Types in Recon

Record Notation has six primitive data types: text, data, num, boolextant, and absent; and one aggregate data type: record. Below, I’ve included a brief description of each data type below. You can read more about Recon data types here on GitHub.

  • Text values: Text values take one of two forms: a quoted string, or an unquoted identifier.
  • Data values: Binary data encodes as a leading '%' symbol, followed by a base64 literal.
  • Num values: Numbers serialize as decimal literals.
  • Bool values: Booleans are represented by the true and false identifiers.
  • Extant values: Extant symbolizes a thing that is defined, but which has no specific value. Extant is represented by an empty token where a value is expected.
  • Absent values: Extant symbolizes a thing that is defined, but which has no specific value. Extant is represented by an empty token where a value is expected.
  • Record values: Recon has a single aggregate data type, called record. Records play the combined role of array and associative array. Think of a record as a partially keyed list—a sequence where some items may have keys, and other items may lack keys. An array is a record in which no items have keys. An associative array is a record in which every item has a key. An object is a record where every item has a text key.

For users trying out Recon for the first time, I recommend starting with the swimOS tutorial. You can also read more about Recon on the swimOS developer site and on GitHub.

Learn More

Let us know what you're building using the open source swimOS platformYou can get started with swimOS here and make sure to STAR us on GitHub.

Topics:Stateful ApplicationsSWIM AIdistributed computingserverlessHTTPweb applicationsswimOSRESTWARPstreamingoperating systemsapistreaming apijavaopen source softwaremicroservices