SQLstream's Relational Asynchronous Messaging (RAM) architecture delivers high performence, resilient and distributed stream computing applications, using an intuitive and powerful streaming SQL platform. The following sections provide an overview of the architectural concepts and features of SQLstream's stream computing architecture:
| RAM – Relational Asynchronous Messaging |
Relational Asynchronous Messaging (RAM), with foundations in the world of Relational Databases, offers a new and fundamentally better way of managing and processing the dynamic data that flow across an enterprise. Dynamic data include application and business transactions, database and file updates, application inputs and outputs, and any of the data feeds processed or generated within an enterprise. Representing data streams as time-stamped messages, RAM processes them using familiar relational operators augmented to handle time windows. |
RAMMS – (RAM Management System) – the analogue to RDBMS |
SQLstream is the world’s first RAMMS, enabling you to create, update and manage RAM applications without pausing execution. The RAMMS is somewhat analogous to the familiar RDBMS of the relational database world. However, RDBMS such as Oracle™ and DB2™ allow easy management and reuse of historical business data through static, stored data with single-shot queries. By contrast, SQLstream's RAMMS uses standing queries that provide easy management and reuse of the dynamic, flowing data assets throughout an enterprise. |
| Publish and Subscribe | RAM can be viewed as a relational Publish/Subscribe technology. When a message is published to a stream, it is delivered to and processed by all queries subscribing to that stream. Queries can even be used to define further streams, which in turn can have their own subscribers. |
| Quality of Service | Inherent in the RAM model is an explicit desired “Quality of Service,” defining whether latency should be minimized, throughput maximized, or reliability levels employed in message delivery. (Examples of reliability levels include “At Least One”, “At Most One,” or “Exactly Once”). |
| Query the Future® | SQLstream's SQL continuously examines and processes incoming data in real-time with minimal latency, and outputs its results as relational message streams to downstream consumers. If a relational database can be viewed as querying past historical, stored data, then SQLstream should be viewed as running queries over future, flowing data. |
| Data-driven, not Query-driven | SQLstream's processing is data-driven. Data arrive and are routed to the relevant SQL processing operations that await them. The outputs are similarly routed to further downstream SQL processing operations. The processing is continuous, and the SQL queries potentially run forever handling flowing data, in sharp contrast to the RDBMS model, where queries repeatedly run to completion against stored data. |
| Dataflow Architecture | SQLstream offers an inherently parallel and distributed processing paradigm known as Dataflow Architecture. In such a system, data flows along arcs connecting a network of nodes where the data are processed. Each node performs relational processing, such as joining data streams, filtering data streams, or composing new record streams (projecting out specific fields, for example). The processing can proceeed in parallel over a distributed network of servers, each processing node outputting data as soon as its requisite input data are available for processing. |
| Pipelined Processing | The data cascade from one node to the next, each performing a valuable step in the overall processing. The cascade forms a pipeline enabling maximum throughput by time-overlapping the processing performed by each step. For example, suppose an overall processing operation, normally performed as a single step taking ten seconds, were instead broken down into, say, ten steps of one second. The potential from pipelining the steps is to produce an output every second, providing a 10X improvement in throughput. |
| RAMPs, Control Nodes, and XMI | All message processing in SQLstream is performed on commodity hardware servers. Such message processing nodes are known as Relational Asynchronous Message Processors (RAMPs). RAMP and metadata management is orchestrated centrally, with a fault-tolerant Control Node that configures and manages the RAMPs. Control Nodes support XMI meta data standards. |
| Efficient Binary Representation | In SQLstream, all data records are implicitly self-describing based on a schema, and all data are held in a highly compact binary representation. These facts enable extremely fast processing by SQLstream queries and operations, minimizing the amount of memory required to hold the messages as they are processed. |
| Chains and Trains | Messages can be chained together in blocks, akin to the carriages of a train, with significant improvement in efficiency. Operations that would have high overhead if performed on a single message, such as network transmission, can be accomplished more efficiently. The result is again maximizing throughput without adding to latency, avoiding the need to constrain maximum time in order to wait for data arrival. |
| Distributed GRID Processing | The dataflow processing model naturally lends itself to implementation over distributed network-connected servers where. SQL queries are broken down into component operations processed at nodes distributed over the network. SQLstream's Query Optimizer is able to transform SQL queries into sub-queries for efficient distributed execution, offering a declarative GRID-processing model. Such processing can be automatically tuned and optimized to take advantage of available hardware. This capability enables dynamic replanning and re-routing of query processing, nimbly avoiding any system outages, and maintaining system and application availability. |
| Self-Healing and Self-Optimizing | SQLstream has the system-wide ability to replan queries and remap them onto available hardware resources without suspending or corrupting execution. This capability provides an extremely powerful and elegant self-healing and self-optimizing data processing infrastructure. |
| Declarative Paradigm | Using pure, declarative SQL to describe all relational stream processing enables RAMMS users to focus on defining data processing requirements rather than specifying how processing is to be done. An optimizer can then use those declarative specifications to determine the most efficient way of executing the SQL while guaranteeing correct processing. (Correctness-preserving, mathematically sound query transformations are used). |
| SQL ANSI Standards Support | SQLstream provides a compliant subset of ANSI standard SQL:2008 (which supercedes our existing support for SQL:1999 and SQL:2003). It extends this standard only by adding the notion of a stream of records rather than just a table of records. |
| Extensible through Plug-Ins | SQLstream provides a number of dynamically installable plug-in adapter modules, written in Java. These adapter modules enable easy and dynamic (i.e., enhancing a system while it runs) extension of the built-in SQL operators. Using these modules, application developers can add new methods of receiving input data, generating output data, transforming streams of data, or operating on individual records. These modules are loaded and plugged into an existing processing network and are managed by the Control Node. |
| Enterprise and Carrier Grade | SQLstream is implemented using state-of-the-art design principles proven through decades of evolution of RDBMS engines such as Oracle™, DB2™ and others. It is engineered to offer the full set of facilities and capabilities one might expect in managing a relational dataset. These capabilities include fully standards-compliant SQL at a deep level: syntax, semantics, null behavior, validation, DDL, DML, security, data-type and schema support.
The code has been hardened and tested against the most stringent correctness and performance requirements. The system has been architected with insights gleaned from SQLstream engineers, who help construct many of the world's leading relational engines. |
| Eclipse Based User Interface | In line with its principle of embracing standards, SQLstream delivers its management and development graphical user interface as a plug-in to the de facto industry standard Eclipse IDE. This immediately familiar interface enables users to be instantly productive. |
| Comprehensive Set of Adapters | SQLstream delivers, out of the box, a comprehensive set of adapters, again supporting relevant ANSI SQL standards such as SQL/MED (mediation of external data). These adapters are packaged as plug-ins that can be dynamically loaded and extended, including JDBC/ODBC drivers, log file parsers, database adapters, JMS and SOAP adapters, email adapters, and many more. |
Have questions about your current project? Click here to ask a SQLstream expert.