Real-time Traffic Apps built on a stream processing platform

Apps have been the single most important IT innovation of recent times. Apps bridge the gap between users and data, providing immediate access to information when it’s most convenient, presented in a way that’s accessible and useful. However, app development, and in particular the backend platform capability, has struggled to keep up with user expectation. Remember the days of the first mobile Internet based on WAP technology? A lesson in over-hype and expectation mismatch.

Traffic apps have been leading the way in app uptake, helping travelers and commuters plan (and replan) their journeys. Yet how many times have you used an app to plan your route, selected the option with the shortest journey time and the least congestion, yet you still arrive late? That’s precisely the problem that Transport for New South Wales (TfNSW), the Australian government agency responsible for managing the road and public transportation networks in Sydney and the New South Wales (NSW) region, Australia, has been addressing in partnership with SQLstream.

Driving Innovation

The initiative began in late 2014 as a code Hackathon for real-time traffic app development. Of the 50 or so teams entered, eight of the most promising and innovative ideas were selected by TfNSW and sponsored through to launch early in 2015. The first five apps have been launched, with more to follow, each focusing on different aspects of the traveler experience. Go to the website for a full description of available apps and App Store download links.

Real-time Traffic App - End to End Journey Time

Real-time Traffic App – End to End Journey Time

  • Snarl gives up-to-date information about road incidents and congestion across the road network and close to the customer’s travel route or location.
  • TripGo is for customers who want to find the most efficient journey from A-to-B using road and public transport options and designs smart itineraries around information on home, work and calendar events.
  • Auto is for customers who have a number of locations they often travel to and want to see the current travel time to help them plan when to leave.
  • Plan A helps customers with a busy schedule who want up-to-date information on congestion, suggested travel times and want to be alerted when to leave to ensure they arrive on time.
  • EasyDrop is for commercial delivery drivers who want an interactive tool to manage their daily delivery schedule, plan for the fastest routes and be up-to-date with traffic conditions.

So if you’re in the New South Wales area, and you’re trying a new app that’s now offering accurate, real-time updates, and suddenly you start arriving on time as planned, it’s likely you are using one of TfNSW’s new road apps that are powered by SQLstream’s stream-processing enabled, real-time data hub and Big Data applications.

The IT Architecture for the Next Generation of Real-time App Platforms

The next generation of web app platforms must scale to deliver accurate, real-time information to a rapidly increasing app community, and scale to support many apps on the same core platform architecture. Conventional web architectures offer on-demand access but can only deliver results based on historical content that is database-driven. As has been the case with traditional enterprise data management architectures, a data stream management platform offers real-time scalability for larger data management applications.

That’s not to say that existing data warehouses are redundant, far from it. The future is an integrated, unified data management architecture that utilizes the strengths of the best mature technologies.

Real-time Data Hub for Web Apps Architecture

The major components of the enterprise web architecture supporting the Travel Time Data System (TTDS) as deployed by TfNSW is as follows:

  • Data Warehouse. The master record is maintained in the data warehouse(s), in this case SQLServer, and is the platform for data mining, trend analytics and enterprise reporting. The SQLServer data warehouse is populated from a number of sources, including existing enterprise systems, as well operational traffic data and streaming congestion analytics streamed through in real-time from the data stream management platform.
  • Real-time Data Hub for Enterprise Apps. Real-time operational intelligence platform is built on SQLstream Blaze with the SmartCity StreamApp. Operational users gain actionable insights through real-time map-based dashboards and alerts. Analytics are the result of the streaming joins of input vehicle GPS and sensor data streams from over 10,000 vehicles, joining with the historical trend data from the SQLServer data warehouse.
  • Scalable Web API Platform for Real-time Access. Web application access for Smartphone apps is provided through a REST API. The API supports real-time queries for journey time updates, traffic flow and incident information as well as crowd-sourced. Additional web servers are added to the cluster as app uptake increases.
  • Real-time Middleware. Inter-application and distributed server cluster scalability requires a middleware-based architecture (Apache Kafka).

Real-time Traffic App API

App developers have access to the Mobile Application Streaming API which is based on HTTP 1.1. The API connects to SQLstream’s Real-time App Platform and supports the following HTTP requests. The API is serviced based on the current state of the road network, traffic flow and incidents, and the current calculations for journey travel times.

Config API:

Returns API URIs and client configuration values.

Route API:

Accepts end to end journey route path and departure/arrival time, and returns the route travel time and current incidents on that path.

Progress API:

Accepts device position, speed and heading and returns travel time updates and current incidents on that path.

Events API:

Returns all incidents on the road network.

Third party developers also have access to the Server API for developing their own on-premise or cloud-based SQLstream applications. The Server API delivers the real-time traffic data feed over a federated connection to the third party SQLstream Blaze system.

Real-time Travel Time Calculations

Real-time Dashboards for Traffic Apps
Journey time planning with real-time updates is the most popular category of traffic apps. Accurate journey times must be calculated based on the actual traffic on the road at that point in time. This is where most device and web apps fall down. Using crowdsourced data and historical trend information alone is insufficient. The Travel Time Data System takes a different approach based on a combination of vehicle GPS data feeds and crowdsourced app data, with traffic flow and congestion alerts calculated in real-time over the entire road network.

The SmartCity StreamApp includes real-time traffic flow and congestion analytics dashboards that are available for internal use, showing the entire road network on interactive, map-based dashboards (click on the thumbnail above for an example). However, app developers are now able to offer travelers access to the same data through their smartphones. The following example is the web-based version (link to of Snarl’s app for up-to-date information on road incidents and congestion across the road network.


Snarl’s Real-time Traffic App (Web Version)

The road network is represented as a GIS database of 10 meter road segments, with a total of 80 million road segments for the New South Wales network. The SQLstream platform updates traffic flow and congestion incident predictions over road segments simultaneously and in real-time for every new data record, where the average current speed for each road segment is calculated as follows.

  • End to end journey times are calculated as a function of all 10 meter road segments in an end to end route.
  • Vehicle GPS data is used to calculate average speed where a vehicle has travelled across that road segment in the past 15 minute.
  • If no vehicle has travelled over a road segment in the past 15 minutes, the 28 day historical average for a comparable 15 minute time period is used for that road segment. That is, the next best choice for Saturday at 9am would be the average for that 9am / 15 minute period for all previous Saturdays within a 28-day window.
  • In the unlikely event that no vehicles have been recorded over a road segment in the past 28 days, the posted speed limit is used.

The minimum and maximum travel times vary based on the above calculations and the variance. The variance is calculated where at least two vehicle reports have been reported over any 10m road segment.

The real-time updates are streamed to both the operations centers, where the current traffic flow and predictions alerts are displayed on color-coded, map-based displays, and also used to service requests in real-time from the smartphone apps through the Mobile Application Streaming API.

What do these new apps mean for commuters?

The next generation of real-time traffic apps offer commuters with accurate and reliable journey time and other traffic-related information. Moving away from a reliance on historical and crowd-sourced data means greater reliability, accuracy and timeliness. However, the real-time processing required also requires a change in mindset. By extending the traditional enterprise data management architecture to embrace the benefits of data stream management, a real-time platform for app support has been delivered that scales with both data growth and app uptake, to the benefit of all concerned.

Real-Time Analytics with Stream Processing for Operational Intelligence and the Internet of Things

There are undoubtedly several approaches to the way systems deal with real-time data before it is persisted in a database. For example, two of the most common open source platforms for this are Apache Storm and Apache Spark (with its Spark Streaming framework), and both take a very different… Read more →

Real-Time Analytics with Stream Processing for Operational Intelligence and the Internet of Things

There was a lot of interest and discussion at the recent Strata in San Jose around real-time analytics. Seems to be the hot topic, and many have been blogging about it since, including why SQL is a good idea for real-time streaming analytics. I would certainly agree with… Read more →

Stream Processing and Streaming Analytics for Telecommunications

Cibertec accelerates their real-time Big Data appliance with the SQLstream Blaze stream processing platform, enabling Communication Server Providers to benefit from real-time billing, call fraud detection, dynamic service quality management and network performance monitoring.
 … Read more →

Stream Processing and Streaming Analytics for Telecommunications

The potential for Big Data in Telecoms is immense, with the global big data market in the telecom sector growing at a CAGR of 55.24% over the period 2011-2015. Streaming integration and analysis of call data records (CDRs and IPDRs)combined with customer, device, location and network data is at… Read more →


Just about everyone has heard the term Big Data, even my most non-technical of friends. Significantly fewer people outside the industry are quite so familiar with the term “self service analytics”, although most can make a stab as to its meaning. Self-service analytics is not a new phenomenon but… Read more →

The real-time data hub for smart city, intelligent transportation and Internet of Things appliances

SQLstream expands its strategic relationship with Altis Consulting for delivery of streaming analytics and Internet of Things services in Asia Pacific region. SQLstream, the provider of the real-time data hub for Enterprise stream processing, today announced the expansion of its partnership with Altis Consulting, a leading providing of Business… Read more →

Real-time Data Hub + Stream Processing = The Internet-of-Things

The™ data service exchange™ for connected device platforms is a marketplace of web applications that operates on data from connected devices to enable data aggregation, analysis, and an expansive range of value-added services for enterprise customers. The number of developers involved in IoT activities will surpass three million… Read more →

Real-Time Analytics with Stream Processing for Operational Intelligence and the Internet of Things

This past year has seen Big Data technology mature into Enterprise-class platforms capable of delivering value in the largest of organizations. Hadoop storage platforms and stream processing are now core components of the standard Big Data enterprise architecture, with traditional RDBMS and warehouse platforms regaining their position in the… Read more →


SQLstream, provider of the real-time data hub for enterprise grade stream processing, today announced the availability of StreamLab 1.1, the next release of the world’s only intelligent guided discovery, streaming analytics and real-time visualization platform. Enterprises using StreamLab are building and deploying massively scalable real-time dashboards from their log… Read more →

Real-Time Analytics with Stream Processing for Operational Intelligence and the Internet of Things

This week saw the publication of the results for a comparative real-time performance benchmark between Apache Storm and SQLstream Blaze. Using the WordCount example shipped with Hadoop and Apache Storm, we were interested to see just how quickly each could process records.
As it turns out, pretty quickly for… Read more →

Real-time to Action for Stream Processing

SQLstream Blaze outperformed Apache Storm by 113X using the industry-standard WordCount benchmark. For enterprises using Blaze, the result means streaming analytics from Big Data in motion using only a fraction of the servers required by open source stream processing frameworks, and therefore a fraction of the cost. Why… Read more →

Security Internet of Things

Cybersecurity and the Internet of Things are increasingly uncomfortable bedfellows. We’ve blogged before on the the security gaps that already exist as a result of connecting yesterday’s technology to the Internet. A recent article by Colin Wood published on goes several steps further and brings us up to date. The… Read more →

Extreme data stream processing performance

The stream processing paradigm differs from the traditional storage-based data management paradigm with which we grew up. Stream processors are fast as they are in-memory (although this is not unusual these days), process data streams record-by-record as they arrive over time or record-based windows using continuous queries (which never… Read more →


Big Data is also about faster results, streaming analytics and real-time actions (as low as millisecond latency) in the case of stream processing, with faster batch operations in the case of Hadoop (a few hours). Fast data also means a different set of data quality issues – which must… Read more →

Real-time to Action for Stream Processing

There’s certainly a vast range of different IoT API, connection protocol technologies and data formats. At first glance, this makes device to device communication tricky, particularly as the Internet of Things encompasses all vendors and technologies. However, the Internet of Things is not necessarily a direct vendor to vendor… Read more →

Real-time to Action for Stream Processing

Bloor Group has launched its Big Data Ecosystem Directory to help their audience and customers research the most promising and innovative software vendors in the Big Data space. The Directory includes  live demos of the products, covering every aspect of the Big Data Ecosystem, as well as the outputs of… Read more →

September 15, 2014 by in In the News
Real-time to Action for Stream Processing

A large variety of commercial and open source event processing software is available to architects and developers who are building event processing applications. These are sometimes called event processing platforms, complex-event processing (CEP) systems, event stream processing (ESP) systems, or distributed stream computing platforms (DSCPs).
Distinguished analyst Roy Schulte from… Read more →

August 27, 2014 by in Stream Processing

Streaming analytics is the latest thing in Big Data: streaming analytics, simply put, enables organizations to leverage their fast Big Data in real-time, improving operational efficiency and customer satisfaction, while enabling new real-time revenue streams. Alex Woodie takes a look at The Forrester Wave™: Big Data Streaming Analytics Platforms, Q3… Read more →


Customer experience management (CEM) has piqued the interest of the communications industry, particularly as service providers seek innovative solutions to drive competitive advantage. CEM solutions offered by companies like Amdocs, Alcatel Lucent, Ericsson, Nokia and Oracle have evolved in recent years from those that independently focus on individual customer… Read more →

Real-time Big Data Cost of Performance

Data processing technologies perform at different rates, making the Total Cost of Performance a hot topic for feasibility studies concerning Big Data tools including traditional databases, Hadoop and stream processors. The reason is simple. Any storage-based technology must store the data first before they can be queried and processed,… Read more →

July 3, 2014 by in Hadoop
Real-time to Action for Stream Processing

Log analytics has been around for a while but until recently, “real-time log analytics” usually meant
a) slow answers (in many minutes or even a few hours)  and
b) low volumes (data arriving at a few thousand records per second).
Things, needless to say, are changing (could be the recent uprise in… Read more →

Stream Processing and Streaming Analytics for Telecommunications

Customer experience has long been a tricky area for telecommunication service providers, wireless and wireline. Not that the data hasn’t been there, but it’s been distributed across different organizational and data silos, and making use of it has been challenging. In fact, the lack of a complete customer view… Read more →


A distributed data management architecture is an essential requirement for real-time Big Data applications such as managing IoT sensor and machine data payloads. Smart services for IoT applications will require low latency answers, multiple servers and distributed processing for scalability, plus built-in redundancy for resilient, 24×7 operations.
It’s also important… Read more →

SQLstream Blaze support for Apache Kafka merges standards-compliant SQL stream processing with Hadoop Big Data platforms on a single real-time, distributed messaging infrastructure.
SAN FRANCISCO, CA – May 29, 2014 – SQLstream Inc., the leading standards-based stream processing software company, today announced support for Kafka, Apache’s messaging framework. Kafka… Read more →

On Thursday, April 24, SQLstream hosted a webinar exploring the potential of the Internet-of-Things. With a focus on monetization, the event expanded on harvesting real-time value from IoT services, discussing technology requirements, security concerns and likely directions for commercialization.
So what is the Internet of Things? To many, it’s about connected devices,… Read more →

 “The 3Cs of the Internet-of-Things: Conditions, Concerns, Commercialization” will run on April 24, 2014
SAN FRANCISCO, CA – April 17, 2014 – SQLstream, Inc., the Big Data Stream Processors company, today announced that it will host a webinar exploring the potential of the Internet-of-Things. With a focus on monetization, the event… Read more →

SQLstream’s Stream Processor to Deliver the Next Generation of Real-time Gaming Analytics  
SAN FRANCISCO, CA – April 1, 2014 – SQLstream Inc., the Big Data Stream Processors company, today announced that The Workshop has selected SQLstream for real-time gaming analytics. The Workshop, a leading US gaming studio and creator of… Read more →

The Internet of things has been on everyone’s lips (and pen) lately, as the biggest new source of money, customer product innovation, investment opportunities, and Sci-Fi-worthy stories. The truth is the value of the Internet of things only comes from the astounding mass of data it’s bound to produce, and… Read more →

What goes around surely comes back around, which in the world of data is often called lifecycle management. Very few organizations have ever formalized and implemented such a grandiose practice, for only until recently has the concept become seriously doable without great expense.
Eric Kavanagh of Inside Analysis writes about a new,… Read more →

APIs are creating a new age of intelligent machine-to-machine (iM2M) analytics apps aided by SaaS. M2M data is everywhere, but individuals don’t want the flood of data — just the pint they need when they thirst for it. Distinguished market researched Kishore Jethanandani talks about how grossly underutilized data, often… Read more →

With the O’Reilly Media’s Strata conference opening one week ago, a lot of big data companies announced new products and looked for the spotlight. In this review published by CNN Money, distinguished analyst Derrick Harris of GigaOM goes through the most notable events in Big Data and lists SQLstream… Read more →

SQLstream 4.0 brings unparalleled real-time performance, Apache Storm integration, enhanced real-time visualization and new industry-specific StreamApps toolkits to the leading standards-based Big Data stream processor.
SAN FRANCISCO, CA – 11 February 2014 – SQLstream, Inc., the Big Data Stream Processors company, today announced the availability of SQLstream 4.0, the new… Read more →

February 13, 2014 by in Press Releases

January 14 Webinar with EMA Industry Expert, SQLstream CEO, and InfoArmor CTO, Will Share Expert Insights on how Enterprises Can Extract Value from Machine Data with Streaming Analytics
SAN FRANCISCO, CA – January 8, 2014–SQLstream Inc., the streaming Big Data platform for real-time operational intelligence, today announced that CEO Damien… Read more →

San Francisco, CA– SQLstream, Inc., the leading standards-based streaming Big Data analytics platform, announced today that Database Trends and Applications (DBTA) has listed SQLstream as a trend-setting product in real-time Big Data management and analytics. The Trend-setting Products in Data for 2014 honors organizations with a commitment to technology innovation and… Read more →

December 16, 2013 by in In the News

SQLstream StreamApps are fast-start templates for real-time streaming machine Big Data applications.  Each StreamApp is a library of components for a specific operational business process. In a Big Data industry typified by a lack of standards and high development costs, StreamApps takes SQLstream’s standards-based SQL platform for streaming operational… Read more →


SQLstream was named as one of the top trend-setting technologies for data management for 2014 by the Database Trends and Applications (DBTA) magazine.
Two of the most pressing concerns that organizations face today are the need to provide analytic access to newer data types such as machine generated data, documents and graphics,… Read more →

December 6, 2013 by in In the News

Just back from the Silicon Valley Comes to Oxford (SVCO), an invite-only event at the Said Business School, University of Oxford University in the UK. The aim of the event is to provide insight to the Business School graduates on how to start, scale and run high-growth companies. The speakers… Read more →

SAN FRANCISCO, CA – 14 November 2013 – SQLstream Inc., the streaming Big Data platform for real-time operational intelligence, today announced StreamApps, fast-start templates for real-time streaming machine Big Data applications.
Each StreamApp is a library of components for a specific operational business process. In a Big Data industry typified… Read more →

Veracity and SQLstream team up to deliver actionable Operational Intelligence for real-time call rating, performance management and fraud detection from streaming CDR and IPDR data
SAN FRANCISCO, CA – 12 November 2013 – SQLstream, Inc., the Streaming Big Data Company, today announced that Veracity Networks, an award-winning internet and telecommunications… Read more →

The real-time data hub for smart city, intelligent transportation and Internet of Things appliances

Australia’s most advanced real-time traffic information system based on SQLstream’s real-time operational intelligence platform listed as key product by Ventana Research in Technology Innovation Award. SQLstream Inc., the streaming Big Data platform for real-time operational intelligence, today announced the award-winning collaboration between Australia’s Roads and Maritime Services and SQLstream…. Read more →

Click on image to listen to podcast
On October 4, 2013, CEO Damian Black briefed members of the Boulder Business Intelligence Brain Trust (BBBT) on the topic “From Big Data to Real-time Value: Streaming Operational Intelligence.”
Esteemed analyst Claudia Imhoff interviews Damian in a short podcast on Big Data, SQL and the… Read more →

On Friday, October 14, 2013, Damian Black, CEO, and Ronnie Beggs, Vice President of Marketing, of SQLstream, Inc. briefed members of the Boulder Business Intelligence Brain Trust (BBBT) on the topic “From Big Data to Real-time Value: Streaming Operational Intelligence.”
Distinguished BI Expert Paul te Vraak comments on the SQLstream… Read more →

October 10, 2013 by in In the News

SQLstream s-Visualizer provides active graphical dashboards over streaming time-series data, mapping real time to human time without refresh
San Francisco, CA | October 8, 2013– SQLstream Inc., the Streaming Big Data Company, announced today the world’s first visualization dashboard product for streaming high-velocity data. SQLstream s-Visualizer provides active, graphical insights… Read more →

SQLstream to Demonstrate Real-time Big Data Solutions for Traffic Analytics and the Internet of Things in the Big Data and Business Analytics Industry Showcase
San Francisco, CA | September 23, 2013 – SQLstream Inc., the streaming Big Data engine for real-time operational intelligence and a Silver level member in Oracle… Read more →

September 24, 2013 by in Press Releases + ,

Distinguished analyst Robin Bloor has long been discussing the term of “data science,” with posts ranging from explanatory to complaining (particularly about the use of terminology).
Throughout all, however, he recognizes data science as a sensible activity for a business to pursue.
In his last post, he is promoting that idea,… Read more →

SQLstream Explores the Total Cost of Streaming Big Data Solutions at TDWI Solution Summit 2013, September 15-17 — CEO Damian Black to Present “Operational Intelligence from Machine Big Data: Case Studies in Taming the Cost of Real-time Performance”
San Francisco, CA, September 4, 2013 — SQLstream Inc., the streaming Big… Read more →

Streaming Hadoop HBase Case Study: Turbo-charging Hadoop for real-time machine data analytics. Session to be Presented at NoSQL Now! on August 22nd at 9:30 PST, San Jose.
San Francisco, CA | August 19, 2013 – SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today… Read more →

Ventana’s Technology Innovation Awards showcase “advances in technology that contribute significantly to improved efficiency, productivity and performance of the organization.” SQLstream’s IT Analytics and Performance award recognized SQLstream’s innovative technology and ability to optimize operational processes and systems. The award considered all aspects of SQLstreams technology and business approach,… Read more →


Ventana Research, the leading benchmark business technology research and advisory services firm, has selected SQLstream Inc. as the winner of its Ventana Research Technology Innovation Award for IT Analytics and Performance. The Ventana Research awards distinguish those pioneers that have developed clear visionary and transformative technology, and that are… Read more →

August 15, 2013 by in Press Releases

Damian Black, SQLstream CEO, is interviewed by Dataversity prior to his presentation at NoSQL Now! 2013. Speaking on streaming SQL and operational intelligence for Hadoop and HBase, Damian discusses the challenges of accelerating Hadoop to process live, high velocity unstructured data streams, and the emergence of SQL as the… Read more →

August 14, 2013 by in In the News

The definition of machine data covers, not surprisingly, all data generated by machines – servers, applications, sensors, web feeds, networks and service platforms. It covers everything from data centers, telecommunications networks and services to machine-to-machine and the Internet of Things in a device-connected world.
The value of machine data is… Read more →

The number of articles about the Internet of Things [IoT], Machine-to-Machine communication [M2M], the Industrial Internet, the Internet of Everything [IoE] and the like have been increasing since this post introducing an IoT mind map almost a year ago. We all learn from some of them, with some we nod sagely in agreement,… Read more →

August 1, 2013 by in In the News + , ,

Folklore has it that the term ‘Internet of Things’ (IoT) was first popularized in 1999 at MIT to describe the architecture of connected RFID devices. Cisco then looked to define when the IoT came in to being as a concrete entity – defined as the year in which the… Read more →

The recent public release of information on the activities of our Intelligence Agencies makes interesting reading for Big Data professionals. In particular, how these agencies have mastered the Internet, with vast data collection and analytics facilities for monitoring Internet and smartphone activity. A series of news articles described GCHQ’s… Read more →

July 5, 2013 by in Market Views

Webinar Series to Explore Challenges and Solutions for Low-latency Operational Intelligence in the Telecommunications, Cybersecurity and Large-scale Sensor Network Industries
San Francisco, CA | June 27,2013 – SQLstream, Inc., the leading standards-based Streaming Big Data Platform, announced today that it will be hosting a series of educational webinars to take place… Read more →

We participated on the “Architecting Big Data Systems for Speed” panel at E2 Conference. Great event, and a great opportunity to discuss technology in a business context. The panel offered a range of perspectives with other panelists from Translattice and Oracle’s NoSQL division. A number of interesting topics emerged,… Read more →

June 19, 2013 by in Streaming Analytics
GigaOM: Structure 2013 Logo

“The Total Cost of Performance in a Massively Connected World” Session to be Presented at GigaOM Structure on June 19th at 4:55PM PST in San Francisco
San Francisco, CA | June 18, 2013 –
SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today announced that SQLstream… Read more →


Ronnie Beggs Participating on “Big Data: Architecting Systems at Speed ” Panel at E2 Conference, June 18th at 2:30PM ET, Boston
San Francisco, CA | June 17, 2013 – SQLstream Inc., the streaming Big Data platform, today announced that SQLstream has been invited to speak at E2 Conference on the architectural… Read more →

Sensors Expo is the leading industry event for the types of intelligent sensor-integrated systems that are driving the next generation Internet of Everything, Industrial Internet, telematics and Machine-to-Machine services. As a key sponsor in the Big Data and Wireless Systems pavilion, and speaking in the Big Data track, we found… Read more →

SQLstream was named as one of the top 100 companies that provide technologies and services for data management by the Database Trends and Applications (DBTA) magazine.
A result of lengthy research and compiled by the publication’s editorial staff, the first “DBTA 100: The Companies That Matter Most in Data” list recognizes industry… Read more →

InfoArmor and SQLstream Team Up to Deliver the Next Generation of Continuous Identity  Monitoring Services at Massive Scale 
SAN FRANCISCO – JUNE 5TH, 2013 – SQLstream, Inc., the Streaming Big Data Company, today announced that InfoArmor, a U.S. leader in identity theft monitoring and internet surveillance, selected SQLstream as the… Read more →


San Francisco, CA | May 30, 2013 – SQLstream Inc., the Streaming Big Data Company, announced today that SQLstream VP Americas, Glenn Hout, has been invited to speak on real-time operational intelligence and prescriptive analytics for the Internet of Everything at Sensors Expo 2013, Jun 4-6.
Held in Rosemont, Illinois,… Read more →

The Internet of Everything is the new frontier for real-time and Big Data, where Velocity now trumps Volume as the primary driver, where the geographical distribution of streaming data adds new levels of complexity, and yet the useful lifetime of data, the window within which to make a decision,… Read more →

Real-time to Action for Stream Processing

SQLstream sponsored the recent IE Group Big Data Innovation Summit in San Francisco where I also presented on streaming SQL for Hadoop, and extending Hadoop for real-time operational intelligence and streaming analytics. As Big Data technologies and Hadoop push further into mainstream enterprises, so the need for real-time business… Read more →


Unlike traditional relational database management systems (RDBMS), which persist data that can later be queried, SQLstream runs continuous queries on targeted data streams – enabling a real-time view of an organization’s situation.
Eric Kavanagh recaps SQLstream’s last session in The Briefing Room, held alongside analysts Mark Madsen and Robin Bloor-… Read more →

April 9, 2013 by in In the News

San Francisco, CA | February 20, 2013 – SQLstream, the provider of the real-time data hub for stream processing applications, today announced an extension to its partner program that helps to fulfill the growing need for real-time data processing systems in supercomputing centers and academic cyberinfrastructure facilities. The SQLstream… Read more →

February 20, 2013 by in Press Releases

Today’s edition of Information Management’s DM Radio Broadcast, The Future of Integration: ETL, CDC and IOA, had a great panel line up discussing the breadth of data integration issues in today’s world of Big Data, Cloud and traditional enterprise architectures. The session was hosted by Eric Kavanagh (Bloor) and… Read more →


Bloor Group’s Robin Bloor hosted SQLstream’s CEO Damian in The Briefing Room in a webcast entitled “Windows of Opportunity: Big Data on Tap”. The webinar focussed on the emergence of both SQL and the stream processing as a key enabler for real-time Big Data systems in an ever-maturing marketplace…. Read more →


Dylan Janus from Information Today talks about the latest SQLstream release on DTBA, and how SQLstream s-Server provides faster response to operational Big Data. The flagship SQL stream processing platform introduces high performance distributed stream processing, with Google BigQuery integration, enhanced platform manageability and enhanced support for streaming analytics…. Read more →


The SQLstream Briefing Room webinar with Robin Bloor “Windows of Opportunity: Big Data on Tap”  highlighted how the evolving Big Data landscape needs technologies that enable a much bigger enterprise-wide picture, complete with multiple data streams that can be combined to show what’s happening in real-time. The speakers were… Read more →

Streaming SQL for Stream Processing and Streaming Analytics

I have recently researched Infosphere Streams’ “Stream Processing Language” or SPL, after a pretty good talk at a recent SVForum SIG group meeting. As I understand it from the talk, early users of this technology have found that their Version 2.0 Stream Processing Application Declarative Engine (SPADE) programming language has been… Read more →

New SQLstream s-Server 3.0 enables faster response to operational Big Data by tapping into streaming log file, sensor and service data in real-time.
SAN FRANCISCO, CA, December 13, 2012 – SQLstream Inc. (, a pioneer of real-time Big Data, today delivered the new generation of its streaming Big Data management… Read more →

The move over recent years has been towards increasingly distributed processing of data, both in terms of the underlying model and in terms of the processing architectures available. More and more of this data is also streaming and SQLstream is focused on effective access to streaming big data …
Read… Read more →

Real-time Big Data means the streaming, continuous integration of high volume, high velocity data from all sources to all destinations,  coupled with powerful in-memory analytics. It’s a paradigm shift from conventional store and process systems that’s playing well here at the Intelligent Transportation Systems and Solutions World Congress here… Read more →

The latest installment in the datacentre mysteries – in which Hadoop has long been the prime suspect for writing off the relational database (RDB) – is out now. With fox-like cunning, [the NoSQL movement] devised a plan so that traditional databases could be slowly migrated onto this new model, without… Read more →

October 20, 2012 by in In the News

We’re at the Intelligent Transportation Society’s World Congress in Vienna, Austria next week, Mon 22 – Fri 26 October. The theme for this year is ‘Smarter on the Way’ and this is the first year since 2009 that it’s been held in Europe. The full panoply of intelligent transportation… Read more →

Perhaps the highlight of Oracle OpenWorld last week, or at least, the most commented on by attendees at our booth, seemed to be Larry Ellison’s demo of Exadata and Exalytics – querying 10 days or so of stored twitter feeds with the hope of finding the best US athlete… Read more →

We’re going to be exhibiting at Oracle OpenWorld with SQLstream next week. Looking forward to it following some great analyst briefings over the past few weeks and a lot of interest now from  companies looking for a real-time Big Data management platform where they can built there apps using… Read more →

September 28, 2012 by in Events

This week we’re at the Intelligent Transportation Society of California’s Annual Conference & Expo in Sacramento. The conference is focussed on the adoption of advanced technologies to improve traveler mobility and heighten safety in California.  Attendees include specialists from government, industry and academia.
This week we’ll be demoing real-time traffic congestion… Read more →

Streaming SQL for Stream Processing and Streaming Analytics

I am going to discuss a SQLstream application for monitoring traffic flow in real-time. In this application, vehicles with GPS enabled devices transmit vehicle position along with other vehicle information such as speed and engine state. SQLstream receives this information as a real-time data stream and uses streaming SQL… Read more →

Datanami article on SQLstream for real-time Big Data solutions, reporting on a SQLstream application for real-time traffic analytics based on live GPS data.
“Providing real-time traffic updates is a practical and feasible application of the big data analysis technology that exists today. According to real-time streaming big data platform vendor,… Read more →

August 3, 2012 by in In the News

SQLstream, the leading platform for real-time Big Data integration and analytics, today announced it has joined the Google Cloud Platform Partner Program as a Technology Partner with its release of the Continuous ETL connector for real-time Big Data integration with Google BigQuery.
Continuous ETL for Google BigQuery
SQLstream’s continuous ETL is… Read more →

Streaming SQL for Stream Processing and Streaming Analytics

Last year has been an interesting experience as I participated in a number of stream processing and streaming analytics customer projects for SQLstream. Developing these real-time, stream computing projects greatly increased my appreciation for the advantages of an open, extensible and standards-compliant middleware infrastructure.
For example, I needed to implement… Read more →

Glue Conference 2012 , Denver CO, at the end of May was a great conference, well attended, knowledgeable participants and is the only conference I know that looks at gluing cloud and mobile applications together with a developer focus.
There was the usual wave of NoSQL, cloud storage, cloud platforms… Read more →

We’re at ITS America Annual Meeting, National Harbor, Washington DC  this week (see the team in action below), the yearly opportunity for the US intelligent transportation community to get together and discuss how IT and technology can be used to better serve travelers, industry and government. Real-time, the Smart… Read more →

May 22, 2012 by in Internet of Things

This week I’m attending an interesting conference at UC Berkeley called the “Berkeley conference on Streaming Data”.  The organizers are primarily astronomers and statisticians, but the talks discuss issues and solutions to streaming data problems across a wide selection of scientific areas and engineering applications.  Real-time streaming analytics and Big… Read more →

Joining real-time structured and unstructured data feeds for better accuracy and reliability from your operational intelligence, and the Text Analytics Summit, 2012, London.
Three IT trends have emerged over the past year – Big Data, real-time and the importance of unstructured data. Taking the latter first, there is an increasing… Read more →

Last week SQLstream sponsored and CEO Damian Black presented at Structure Data in New York, a conference exploring “the technical and business opportunities spurred by the growth of big data”.
It’s clear that Big Data has moved on considerably in a very short space of time. From the Silicon Valley,… Read more →

GigaOM reports on Damian Black, SQLstream CEO, talking about streaming Big Data at Structure Data, New York,March 21 – 22. In the talk entitled “Streaming Big Data: Millions of events per second”, Damian discussed the similarities between Hadoop Map/Reduce and the parallel, distributed architecture used for streaming data processing… Read more →

March 26, 2012 by in In the News

For Big Data, 2012 has started where 2011 left off, with a plethora of reports, articles and blogs. Interestingly, most still begin with the question “what is Big Data”. It appears ‘Big Data’ as a market is broadening its footprint far beyond its open source and Hadoop origins…. Read more →

January 11, 2012 by in Hadoop

The Tutorial blog series helps SQLstream developers build streaming SQL applications. This blog is the third and final part of the Geospatial Visualization tutorial. The first blog in the series set out the streaming use case for connecting SQLstream to a Google Earth visualization, and described the initial steps… Read more →

Streaming SQL for Stream Processing and Streaming Analytics

The Tutorial blog series helps SQLstream developers build streaming SQL applications. This blog is the second in the Geospatial Visualization tutorial.  The first blog in the series set out the streaming use case for connecting SQLstream to a Google Earth visualization, and described the initial steps required to capture… Read more →

A defining feature of the show is the Technology Showcase featuring  demonstrations from some of the technologies and applications that are bringing the future of transportation to life.  Each ‘village’ covers a specific theme such as Safety, Mobility, Environment/Sustainability and Pricing.  Environment/Sustainability focuses on the potential for reducing emissions. … Read more →

SAN FRANCISCO, CA, October 17, 2011 - SQLstream Inc. today announced the public availability of SQLstream ITS Insight, the first real-time solution for reducing congestion to exploit low cost wireless GPS data as a complement to existing fixed-road sensor investment. Transportation Agencies are already benefiting already from SQLstream ITS Insight,… Read more →

Visit SQLstream on Booth #1366
Technology and innovation are central themes of this year’s ITS World Congress.  There’s been much written about the issues of congestion, green transportation schemes and improving personal mobility, not least in this blog.  At SQLstream we’ve been playing our part to help revolutionize the Intelligent… Read more →

Streaming SQL for Stream Processing and Streaming Analytics

A streaming SQLstream application will feel very familiar to anyone with some basic knowledge of SQL and traditional RDBMS applications.  SQLstream uses standards-based SQL, except that streaming SQL queries run forever, processing data as they arrive over specified time windows. This blog is the first in a series of… Read more →

The 18th World Congress on Intelligent Transport Systems (ITS) is being held in Orlando from October 16th – 20th, 2011. This is the leading event for intelligent transportation solutions, and attracts a large audience of government, technology and industry professionals. The event seeks to demonstrate advances in the application… Read more →

The latest product update of SQLstream, version 2.5.1, has just been released and shipped.  This will be the final 2.x release prior to the SQLstream 3 launch, and although SQLstream 2.5.1 is predominately a maintenance release, it does include a range of feature enhancements, including:
- Support for exponentially decaying… Read more →

August 10, 2011 by in Streaming Analytics

SQLstream is helping to predict earthquakes across the world in real time. The system has been developed by a consortium of universities and government agencies, with funding from NSF (National Science Foundation), to provide an infrastructure of networked tools for research in ocean science – constructing an internet-based system… Read more →

SQLstream has been powering Mozilla’s Firefox Download Monitor since 2009. A SQLstream based application has been continually aggregating hundreds of millions of download events, receiving minute by minute aggregations via a continuously running SQL SELECT statement using the SQLstream JDBC driver. A continuously running SELECT statement is syntactically and… Read more →

Attendance at the PostgreSQL West 2010 Conference was encouraging considering a million people had gathered in the city to celebrate the World Series victory of the San Francisco Giants.
I’ve posted the presentation from the event (previously blogged about here).
We presented the concepts of Streaming GIS, integrating SQLsteam’s real-time streaming… Read more →

SQLstream’s founding engineer Sunil Mujumdar is set to present at PostgreSQL Conference: West 2010.
In a talk entitled ‘Streaming GIS using PostGIS and SQLstream’, Sunil will describe the SQLstream stream computing platform based on industry-standard SQL, and its integration with the PostgreSQL-based Geographic Information Systems (GIS) engine PostGIS.
Wireless sensors and… Read more →

A streaming SQL query is a continuous, standing query that executes over streaming data. Data streams are processed using familiar SQL relational operators augmented to handle time sensitive data. Streaming queries are similar to database queries in how they analyze data; they differ by operating continuously on data as… Read more →

In the game industry, complex game logic needs to be applied to streams of events generated by gameplay.  In single player games, this logic is simply handled by applying the correct computations.  However, in an Internet based social game where millions of players interact together online, the problem takes… Read more →

Businesses need to respond faster than ever to customer information and demands, which are arriving in rapidly increasing volumes from ever more diverse and distributed systems. This need for real-time business models can not be addressed by traditional integration and business intelligence solutions because streaming analytics and related concepts… Read more →

Just back from the 2010 Intelligent Transportation Society of America’s Annual Meeting.  For those unfamiliar with intelligent transportation, I am not referring to the “shovel ready” projects that have been funded by President Obama as part of the economic stimulus package. These projects were designed to spend money and… Read more →

Contact Us

Would you like to know more about stream processing?

+1 877 571 5775

Ask a Question