Codeworks Hackathon for Real-time Traffic Apps - Powered by SQLstream

Roads users in NSW, Australia are now one step closer to having even better information on their journey at their fingertips.  The winners of NSW Codeworks Hackathon have now been announced and the official video of the event posted. The Hackathon was organized by Transport for New South Wales (TfNSW) and Deloitte with the objective to delivering innovative apps for improving traffic congestion around specific hotspot areas in the Sydney and NSW road network. Powered by SQLstream, the teams were selected for the event, had access to real-time aggregated traffic information based on the SQLstream TT5 platform already in operation with the Australia government.

Powered by SQLstream: NSW Codeworks Hackathon for real-time Traffic Apps

Eight winning teams were selected from the 27 teams selected for the event. You can read more here, but for now, here’s the list of the teams names and the ideas:

  • Toastedmint – Provides transport options to determine the quickest and easiest routes for road users.
  • SkedGo – Provides real-time road information to the public in the context of their trip with calendar integration.
  • Hot Chocolate – A feature rich application that allows users to enjoy an end-to-end road journey with calendar integration.
  • Compiling – Compare multiple transport options to determine the fastest and easiest way.
  • Appjourney – Select the optimal route to avoid congestion and incidents.
  • Transnotion – Provides an effective way for parents to car pool.
  • Somecat Studio – Helps users find a better departure time or mode of transport for reduced travel time.
  • OzPoint – Provides delivery drivers with a trip planning solution.

Our congratulations to all involved. A great event which will make a real-time impact on the NSW traveler experience.

October 17, 2014 by in Internet of Things
Security Internet of Things

Cybersecurity and the Internet of Things are increasingly uncomfortable bedfellows. We’ve blogged before on the the security gaps that already exist as a result of connecting yesterday’s technology to the Internet. A recent article by Colin Wood published on goes several steps further and brings us up to date. The Importance of Cybersecurity in the Age of the Cloud and Internet of Things discusses how Government IT organizations are losing control of their technology with the move towards the Cloud and BYOD, and away from the secure, internal data center and vetted equipment. Organizations are realizing that cybersecurity attacks will… Read more →

October 16, 2014 by in Internet of Things
Big Data Cloud Streaming Analytics

The stream processing paradigm differs from the traditional storage-based data management paradigm with which we grew up. Stream processors are fast as they are in-memory (although this is not unusual these days), process data streams record-by-record as they arrive over time or record-based windows using continuous queries (which never end), and, importantly, incrementally update all results of all executing queries for each and every new arriving record. Contrast this with the batch-based processing of Hadoop or an RDBMS data warehouse, where queries must be re-executed across the entire dataset in order to update results when new data arrive. This is… Read more →

October 6, 2014 by in Stream Processing & SQL
SQL in Hadoop

The Big Data movement has focussed on technology until recently, but as companies such as Cloudera are pushing Hadoop distributions further into the Enterprise, it’s clear that the traditional issues such as data quality still exist. However, it also seems that extending the Enterprise IT footprint to process unstructured data closer to source and in real-time (or near real-time) has shifted the data quality paradigm.

Enterprises become aligned to the batch-based capabilities of their Enterprise architecture and the relational model for data storage, with a heartbeat of 24 hours to the next report update (typically), and reports that are based… Read more →

September 30, 2014 by in Hadoop & Big Data Storage

There’s certainly a vast range of different IoT API, connection protocol technologies and data formats. At first glance, this makes device to device communication tricky, particularly as the Internet of Things encompasses all vendors and technologies. However, the Internet of Things is not necessarily a direct vendor to vendor issue. Rather the need for communication through the real-time data hubs on which Internet of Things applications are being built. It’s therefore the data hub that provides the device to device intelligence and communication – both read and update.

If we look at other industries, particularly telecoms, the concept of a multi-vendor… Read more →

September 29, 2014 by in Internet of Things
Stream processing - real-time for Big Data

There seems to be a lot of discussion recently around Big Data and real-time. As it emerges that Hadoop is not a real-time as hoped, the discussion has moved on to stream processing. I guess the discussion is not helped by the definitions (or lack of) for real-time system behavior. That is until you enter the world of safety-critical systems and perhaps it is this that skews the discussion of real-time systems. Safety-critical systems tend to have specific operating criteria and deadlines that must be met. For example, flight control software and ABS braking. However, these systems ensure response and… Read more →

September 24, 2014 by in Stream Processing & SQL
Home Automation and The Internet of Things

Or in technology terms, is the Internet of Things anything less than magic? I was explaining the concept of the Internet of Things to some friends, successful business people who only embrace technology once it’s matured and has slipped invisibly into their daily routine. Examples include ABS braking systems on a car, mobile phones, more latterly the Internet, but not as yet Google Glasses. I used the admittedly tired example of home automation and the intelligent fridge, with the recent acquisition of Nest by Google to justify the existence of a market ready for mass adoption. ‘Just like Mary Poppins… Read more →

August 20, 2014 by in Internet of Things
Real-time Big Data Cost of Performance

Data processing technologies perform at different rates, making the Total Cost of Performance a hot topic for feasibility studies concerning Big Data tools including traditional databases, Hadoop and stream processors. The reason is simple. Any storage-based technology must store the data first before they can be queried and processed, and the entire dataset must be requeried when new data arrive. This introduces latency into the system. Adding more firepower in terms of more servers can help to a point, but inevitably a tipping point is reached where it’s simply impossible to reduce the latency further. This is the point… Read more →

Stream processing for real-time Big Data log analytics

Log analytics has been around for a while but until recently, “real-time log analytics” usually meant

a) slow answers (in many minutes or even a few hours)  and

b) low volumes (data arriving at a few thousand records per second).

Things, needless to say, are changing (could be the recent uprise in all-things-smart, or a fair need that business decisions are made as fast as possible, or a combination of both). In any case, recently we’ve been seeing real-time log analytics where the customer requires sub-second latency at data rates varying from 50,000 records per second upwards (the highest was 100 million records… Read more →

Customer experience has long been a tricky area for Telcos. Not that the data hasn’t been there, but it’s been distributed across different organizational and data silos, and making use of it has been challenging. In fact, the lack of a complete customer view has driven partial insights based on limited KPIs, resulting in a view of the world that’s always amber, there’s always something a little bit wrong; the information content is not very useful at all.

Understandably, the focus has been on the corporate and VIP accounts.  Here the issue is often simpler as there is likely to be… Read more →


A distributed data management architecture is an essential requirement for real-time Big Data applications such as managing IoT sensor and machine data payloads. Smart services for IoT applications will require low latency answers, multiple servers and distributed processing for scalability, plus built-in redundancy for resilient, 24×7 operations.

It’s also important to offer a range of options for resilience and for implementing the distributed pipework between servers. Customers often have their preferred products and prefer where possible to reuse expertise and to manage their distributed operations on the platforms in which they’ve gained confidence. Different infrastructure tools also have different strengths… Read more →

Security Internet of Things

The Internet of Things has been on everyone’s minds lately, and as new applications are being developed, new revenue streams identified, and new audiences attracted into the faster, more convenient and less wasteful paradigm, the question remains: can we make it work? Apart from the technology issue (you know, that little detail of data becoming too Big to handle), there are plenty of concerns voiced by various commentators.

The ethics of owning and handling personal data are brought up frequently, as courts have yet to rule on what is and what is not private data, but also whether that data, when automatically… Read more →

May 22, 2014 by in Internet of Things

On Thursday, April 24, SQLstream hosted a webinar exploring the potential of the Internet-of-Things. With a focus on monetization, the event expanded on harvesting real-time value from IoT services, discussing technology requirements, security concerns and likely directions for commercialization.

So what is the Internet of Things? To many, it’s about connected devices, where anything that can be connected will be, coupled with staggeringly large estimates for the number of connected devices in the near future as we move towards a connected everything society.

However, classifying the Internet of Things as a world of connected devices, is rather like classifying the Internet as a collection… Read more →

May 15, 2014 by in Events · Internet of Things

Many of the growth charts and predictions for the Internet of Things  originated from a Cisco paper in 2011,  which stated that even by 2008 or so, there were already more connected devices than people on the planet, and that the number of connected devices would grow 25% year on year to get to 50 billion by 2020. More recent predictions from Gartner place Internet of Things growth at 26 billion by 2020, but importantly for the smart services market, breaking down the overall predicted market value for Internet of Things of $1.9 trillion dollars, to estimate a product and… Read more →

May 6, 2014 by in Internet of Things

The “Big Data at Work in the Internet of Things Era” webinar (Tuesday 6 May) explored the potential of the Internet-of-Things, focussed on turning sensor data into business insight for the enterprise, and covered the vision, use cases, success stories and best practices to maximize value from the Internet-of-Things. The webinar was hosted by Oracle, and included Hitachi and SQLstream as guests.

We discussed real-time smart services for the Internet of Things, with a focus on Smart City applications and Intelligent Transportation solutions. In particular, covering the SQLstream partnership with Oracle in their Public Sector and Smart Cities portfolio, and presenting… Read more →

Just back from the Silicon Valley Comes to Oxford (SVCO), an invite-only event at the Said Business School, University of Oxford University in the UK. The aim of the event is to provide insight to the Business School graduates on how to start, scale and run high-growth companies. The speakers include prominent entrepreneurs, innovators and investors from Silicon Valley. Other speakers included Phil Libin CEO of Evernote and Mike Olson, co-founder of Cloudera.

SVCO 2013… Read more →

The Ventana Research Technology Innovation Summit was held this week in Chicago, where VR CEO Mark Smith and BI analysts such as Tony Cosentino presented their perspective on Big Data technology adoption and the issues being faced by organizations as a result. This was not a conference about Hadoop, far from it, in fact the focus was more on how to best exploit Big Data as an asset and how this is being achieved across different software application areas. I would also mention that one of these areas… Read more →

November 15, 2013 by in Hadoop & Big Data Storage

Ventana’s Technology Innovation Awards showcase “advances in technology that contribute significantly to improved efficiency, productivity and performance of the organization.” SQLstream’s IT Analytics and Performance award recognized SQLstream’s innovative technology and ability to optimize operational processes and systems. The award considered all aspects of SQLstreams technology and business approach, in particular:

Vision – bringing standards to the world of Big Data and delivering a platform for real-time operational intelligence and integration across all industries.
Technology innovation – massively scalable, streaming data management platform using standard SQL (SQL:2008) as the query language.
Operational process optimization – enabling business to eliminate latency… Read more →

The definition of machine data covers, not surprisingly, all data generated by machines – servers, applications, sensors, web feeds, networks and service platforms. It covers everything from data centers, telecommunications networks and services to machine-to-machine and the Internet of Things in a device-connected world.

The value of machine data is immense. Machine data contains a wealth of information on customer and consumer behavior and location, consumer quality of experience, financial transactions, security and compliance breaches, as well as the state of industrial processes, transportation networks and vehicle health.

Understanding Machine Data in Real-time

Log files… Read more →

August 12, 2013 by in Hadoop & Big Data Storage

Folklore has it that the term ‘Internet of Things’ (IoT) was first popularized in 1999 at MIT to describe the architecture of connected RFID devices. Cisco then looked to define when the IoT came in to being as a concrete entity – defined as the year in which the number of connected devices exceeded the number of connected people (somewhere between 2003 and 2010 – see diagram).

Click to zoom

Cisco predicted that the number of devices will grow to 50 billion by 2020,… Read more →

July 15, 2013 by in Internet of Things

Sensors Expo is the leading industry event for the types of intelligent sensor-integrated systems that are driving the next generation Internet of Everything, Industrial Internet, telematics and Machine-to-Machine services. As a key sponsor in the Big Data and Wireless Systems pavilion, and speaking in the Big Data track, we found the event a great litmus test for the current state of the sensor-related systems’ industries.

This year, the focus was very much on the monetization of intelligent sensors networks. This included how businesses and consumers can benefit from the Internet of Things, but also a focus on infrastructure costs, in… Read more →


San Francisco, CA | May 30, 2013 – SQLstream Inc., the Streaming Big Data Company, announced today that SQLstream VP Americas, Glenn Hout, has been invited to speak on real-time operational intelligence and prescriptive analytics for the Internet of Everything at Sensors Expo 2013, Jun 4-6.

Held in Rosemont, Illinois, and now in its 27th year, Sensors Expo is the leading industry event in its class and gathers the world’s top engineers, scientists and business leaders involved in the development and deployment of sensor networks and systems. Areas of focus include sensor Big Data analytics, wireless sensor networks for Machine-to-Machine services,… Read more →

The Internet of Everything is the new frontier for real-time and Big Data, where Velocity now trumps Volume as the primary driver, where the geographical distribution of streaming data adds new levels of complexity, and yet the useful lifetime of data, the window within which to make a decision, has decreased dramatically.

Industries such as telecommunications, telematics, M2M and the Industrial Internet are starting to generate high velocity data streams at rates of millions of records per second. Gaining actionable insight from data of this magnitude and speed may be technically feasible to a point for the elastic scale-out architectures of… Read more →

SQLstream sponsored the recent IE Group Big Data Innovation Summit in San Francisco where I also presented on streaming SQL for Hadoop, and extending Hadoop for real-time operational intelligence. As Big Data technologies and Hadoop push further into mainstream enterprises, so the need for real-time business operations is an important parallel trend. ‘Real-time’ and ‘Hadoop’ had been considered synonymous by some, yet surprisingly, people are surprised when Hadoop does not seem to be as real-time as they hoped. This should not come as a surprise, as Hadoop as many strengths, but was never intended for low latency, real-time analytics over… Read more →

April 21, 2013 by in Events · Hadoop & Big Data Storage

Today’s edition of Information Management’s DM Radio Broadcast, The Future of Integration: ETL, CDC and IOA, had a great panel line up discussing the breadth of data integration issues in today’s world of Big Data, Cloud and traditional enterprise architectures. The session was hosted by Eric Kavanagh (Bloor) and Jim Ericsson (ex-Information Management, today was his last DM Radio after 5 years at the helm), and supported by Philip Russom (TDWI). SQLstream’s Damian Black was on the panel, with representatives from Denodo and Dell.

Extending the reach of Hadoop to the edge of your business.

A key observation emerged from… Read more →

I was listening to the BBC News on the Internet, and the technology section had two experts explaining Big Data to the layman. After some mutterings about it being ‘very large’ and ‘useful in genomics’, most listeners would have been left none the wiser. The BBC’s Science Editor, Susan Watts, had a bit of a better crack at it here. Our industry obviously has a way to go in explaining itself to the wider world.

Susan Watt’s BBC article of Big Data

Tackling the idea of ’very large’ first, and how Big is Big Data? Turns out, it isn’t,… Read more →

January 24, 2013 by in Hadoop & Big Data Storage

Bloor Group’s Robin Bloor hosted SQLstream’s CEO Damian in The Briefing Room on Tuesday January 8th 2013. The webcast, entitled “Windows of Opportunity: Big Data on Tap” focussed on the emergence of both SQL and the streaming data platform as a key enabler for real-time Big Data solutions in an ever-maturing marketplace. You can watch the full webinar from the link below, but I’m going to focus on some of the topics arising from the online discussion between Robin, Damian and the audience.

It was an interesting discussion, covering Big Data and streaming in the wider context… Read more →

January 17, 2013 by in Events · Hadoop & Big Data Storage

Dylan Janus from Information Today talks about the latest SQLstream release, 3.0. With improved speed and scalability, the flagship product introduces high performance distributed stream processing. Google BigQuery integration, and enhanced platform manageability and streaming application development.

“We have improved ways of processing machine data, so you can process much higher volumes and much more flexible and resilient processing of log file data,” Damian Black, CEO of SQLstream, tells 5 Minute Briefing.

Read the full article on here. 

Real-time Big Data means the streaming, continuous integration of high volume, high velocity data from all sources to all destinations,  coupled with powerful in-memory analytics. It’s a paradigm shift from conventional store and process systems that’s playing well here at the Intelligent Transportation Systems and Solutions World Congress here in Vienna.

GigaOM took their successful format for the Structure events to Europe this week for the first time and SQLstream’s CEO, Damian Black, was one of the key speakers. In a session with GigaOM’s leading Big Data writer, Derrick Harris, Damian discussed how real-time Big Data can only be achieved using continuous streaming integration, and how standard SQL is the perfect vehicle as a powerful language for the massively parallel distributed processing of live data streams.

As always, the GigaOM events were extremely well executed,and drew many of the main thought leaders and business leaders in the industry. For example, Ame Awadakkah… Read more →

We’re at the Intelligent Transportation Society’s World Congress in Vienna, Austria next week, Mon 22 – Fri 26 October. The theme for this year is ‘Smarter on the Way’ and this is the first year since 2009 that it’s been held in Europe. The full panoply of intelligent transportation issues is addressed, everything to do with software, hardware and firmware that improves the traveler experience, transportation network utilization and capacity optimization. However, the combination of real-time traveler information systems, wireless sensors, GPS and predictive analytics is a specific focus. And increasingly, telematics and V2I/V2V applications are represented.

We’re going… Read more →

Perhaps the highlight of Oracle OpenWorld last week, or at least, the most commented on by attendees at our booth, seemed to be Larry Ellison’s demo of Exadata and Exalytics – querying 10 days or so of stored twitter feeds with the hope of finding the best US athlete from the recent London 2012 Olympics to endorse a car company. This seemed to strike a chord with the audience. How many organizations employ a marketing analytics company to spend a vast amount of time poring over data to work out the top candidates for a marketing campaign? That… Read more →

October 11, 2012 by in Events · Hadoop & Big Data Storage

This week we’re at the Intelligent Transportation Society of California’s Annual Conference & Expo in Sacramento. The conference is focussed on the adoption of advanced technologies to improve traveler mobility and heighten safety in California.  Attendees include specialists from government, industry and academia.

This week we’ll be demoing real-time traffic congestion analytics and visualization based on processing live vehicle GPS data. Play the video below to watch a short video of the live customer case study.

The real-time traffic analytics and connected vehicle programs are of particular interest to SQLstream.  SQLstream turns live data into real-time value, enabling industries such as… Read more →

SQLstream has joined the Google Cloud Platform Partner Program as a Technology Partner with the release of our Continuous ETL connector for real-time Big Data integration with Google BigQuery. Continuous ETL solves the real-time performance issue for Big Data storage platforms. Although very capable of processing large datasets once the data has been stored, Big Data storage platforms are not designed to process real-time streaming data.

Continuous ETL for Google BigQuery

SQLstream’s Continuous ETL connectors enable SQLstream to integrate and analyze vast volumes of live, real-time Big Data, and to update the Big Data storage platforms immediately as input data arrives…. Read more →

We’ve been exhibiting at Structure 2012 in San Francisco, where our CEO Damian Black was speaking on dataflow architectures for massively scalable real-time Big Data computing. In fact, this was a milestone for us as Damian was on the very first Big Data panel at the first Structure event in 2008.

Dataflow is a technique for parallel computing that emerged from research in the 1970s. It’s based on graph-based execution models where data flows along the arcs on a graph and is processed at the nodes. It was decades ahead of its time… Read more →

Glue Conference 2012 , Denver CO, at the end of May was a great conference, well attended, knowledgeable participants and is the only conference I know that looks at gluing cloud and mobile applications together with a developer focus.

There was the usual wave of NoSQL, cloud storage, cloud platforms and Hadoop presentations, as you’d expect, but also with some interesting keynotes as well. Ray O’Brien, CTO for IT at NASA. talked about the evolution of Nebula and OpenStack at NASA, and James Governor from Redmonk, talking about the evolution of… Read more →

This week I’m attending an interesting conference at UC Berkeley called the “Berkeley conference on Streaming Data”.  The organizers are primarily astronomers and statisticians, but the talks discuss issues and solutions to streaming data problems across a wide selection of scientific areas and engineering applications.  Real-time streaming Big Data applications presented included oceanography biology genetics, reading handwriting, astrophysics, particle physics, recommendation engines for social media, and inevitably, real-time fraud detection from live data feeds.

I presented on a deployment of SQLstream as a Dynamically Scalable Cloud Platform for the Real-Time Detection of Seismic Events. Based on work with UCSD seismologists,… Read more →

The Text Analytics Summit in London this week was an opportunity to catch up on the latest trends and state of the Text Analytics market.  An interesting couple of days with a few themes emerging.

Firstly, Big Data.  Not entirely unexpected, but almost every presentation referred to Big Data in some shape or form.  In part this was referring to the volume of data to be processed, but primarily in the context of databases for the storage and processing of unstructured data of any volume.

Although not discussed explicitly, there’s obviously a search for business models that work.  Most applications were… Read more →

Joining real-time structured and unstructured data feeds for better accuracy and reliability from your operational intelligence, and the Text Analytics Summit, 2012, London.

Three IT trends have emerged over the past year – Big Data, real-time and the importance of unstructured data. Taking the latter first, there is an increasing awareness that much of the data we have available to us today is unstructured (Cloudera amongst the many claiming 80% of all data is unstructured).  Unstructured data includes text messages, documents, tweets emails and video content. There’s also a growing industry for tools and software that perform… Read more →

Last week SQLstream sponsored and CEO Damian Black presented at Structure Data in New York, a conference exploring “the technical and business opportunities spurred by the growth of big data”.

It’s clear that Big Data has moved on considerably in a very short space of time. From the Silicon Valley, 101 world of Java developers and Hadoop, into the mainstream wider business world (but still with Hadoop!).

Some themes emerging from the conference:

The basic need to deliver high performance, massively scalable computing infrastructure as data volumes grow exponentially. It’s clear that the pain from… Read more →

Visit our new website to find out more about real-time Big Data applications

Big Data is here to stay. The breadth of the term Big Data may change as it becomes as much a marketing imperative as the ‘Cloud’ word, but the requirement for ‘supercomputing lite’ processing for the non-supercomputing world of enterprise data is a must have.

The rise of Big Data has happened in parallel with the emergence of real-time operational intelligence, and the extension of real-time analytics into the world of real-time updates and process control. Much of the recent interest has focussed on how these two… Read more →

SQL is a declarative language – a SQL query is a specification for the result, it’s neither a recipe nor a program to produce the results. A traditional relational database query returns a set of rows, the ResultSet. A streaming SQL query in SQLstream returns a stream of rows. That is, the ResultSet may never end. In a traditional relational database query, all the rows are fetched, and the ResultSet scans them. With a relational streaming query, the result rows do not exist as yet – as time goes by they come into existence as arriving data are processed.

However, just… Read more →

One of the great advantages of SQLstream as an analytical platform is that it uses the most popular, standardized language for data analysis, SQL. SQLstream worked to make only the minimum number of extensions to SQL necessary to encompass the streaming data paradigm, so that most streaming SQL pipelines look almost indistinguishable from SQL for reading static relational data. This enables data analysts to leverage virtually all of their existing SQL skills in the streaming context.

Similarly, SQLstream felt it was important to make the streaming environment feel familiar and productive to application developers as well, so SQLstream supports the standard… Read more →

February 17, 2012 by in Stream Processing & SQL

QoS and service level monitoring has always presented a challenge for telecommunications companies. With the increase in uptake of IP voice and video services, the vast data volumes generated, and the lack of an end to end view, make monitoring the service experience in real-time increasingly difficult.

In this blog I’m looking at the core building blocks of a real-time IP service monitoring solution by using a much simplified view of a real-time application. Diagram 1 illustrates the basic problem – how to monitor an IP service when the end to end view is only possible by… Read more →

For Big Data, 2012 has started where 2011 left off, with a plethora of reports, articles and blogs. Interestingly, most still begin with the question “what is Big Data”. It appears ‘Big Data’ as a market is broadening its footprint far beyond its open source and Hadoop origins. My favourite new term in this quest for delineation is “Small Big Data”. (Isn’t that just “Data”?)

The most interesting trend for us is streaming Big Data processing and analytics. Edd Dumbill, O’Reilly Radar, talks about this as one of the “Five big data predictions for 2012”,… Read more →

January 11, 2012 by in Hadoop & Big Data Storage

Event report from ITS California,  Annual General Meeting in Long Beach, Nov 13 – 15, 2011.

ITS California’s AGM is SQLstream’s local intelligent transportation event, and an opportunity for public and private sector companies to discuss those issues specific to the state of California. The event has grown in size significantly over the past few years, with new organizations attending, all keen to contribute and discuss different perspectives on the various transportation problems. ITS-CA was established in 1994 as a not-for-profit organization with the remit to foster the adoption of ITS technology across the state. Its funding… Read more →

November 18, 2011 by in Transportation

It struck me that the underlying theme of the conference could be described as ‘low risk innovation’.  An oxymoron?  At first glance, yes, but in this case it describes the cautious adoption of new technology while protecting existing investment.

The perception of the ITS industry, rightly or wrongly, has been one of an industry focussed on major manufacturing and hardware deployment projects, rather than software and new technologies. Therefore perhaps the industry hasn’t seen the level of growth and innovation that has occurred in other areas, telecommunications being a good example.  Or perhaps it simply hasn’t been possible until new technologies… Read more →

October 21, 2011 by in Transportation

A defining feature of the show is the Technology Showcase featuring  demonstrations from some of the technologies and applications that are bringing the future of transportation to life.  Each ‘village’ covers a specific theme such as Safety, Mobility, Environment/Sustainability and Pricing.  Environment/Sustainability focuses on the potential for reducing emissions.  Interesting demonstration offered by Imperial College, London, that will use a pollution sensor mounted on the roof of a demonstration vehicle travelling around the site.  Data is transferred over a GSM connection and viewed over the internet. Another is Ricardo Engineering’s demonstration of improving fuel efficiency using GPS navigation and traffic… Read more →

October 19, 2011 by in Transportation

The ITS World Congress claims to be the largest transportation event in 2011. Certainly the range of attendees and exhibitors is impressive, from software products to the latest in roadside hardware infrastructure. Some interesting themes emerging. Vehicle to vehicle communication is generating a lot of interest, and electronic driver aids. Some great simulators as well to amuse the attendees.

‘Real-time’ is also a common theme across the exhibition hall. And in particular the use of new technologies such as Bluetooth and GPS. Damian Black, SQLstream’s CEO, was speaking in a session today on how… Read more →

October 18, 2011 by in Transportation

SAN FRANCISCO, CA, October 17, 2011 - SQLstream Inc. today announced the public availability of SQLstream ITS Insight, the first real-time solution for reducing congestion to exploit low cost wireless GPS data as a complement to existing fixed-road sensor investment. Transportation Agencies are already benefiting already from SQLstream ITS Insight, using it to deliver real-time Travel Time and congestion detection solutions. The official public launch for SQLstream ITS Insight is today at the ITS World Congress, Orlando.

For commuters in their cars, traffic lines are lengthening and travel times are increasingly unreliable. Congestion is growing globally and transportation agencies are already struggling… Read more →

October 17, 2011 by in Press Releases · Transportation

Visit SQLstream on Booth #1366

Technology and innovation are central themes of this year’s ITS World Congress.  There’s been much written about the issues of congestion, green transportation schemes and improving personal mobility, not least in this blog.  At SQLstream we’ve been playing our part to help revolutionize the Intelligent Transportation industry.  It’s clear that the concepts of streaming data and real-time analytics are entering the main stream – from low level Big Data toolkits that require a streaming, low latency front end, to the real world of sensor networks and industries such as smart grid and telecommunications.

This is… Read more →

A streaming SQLstream application will feel very familiar to anyone with some basic knowledge of SQL and traditional RDBMS applications.  SQLstream uses standards-based SQL, except that streaming SQL queries run forever, processing data as they arrive over specified time windows.

This blog is the first in a series of tutorials for SQLstream developers, describing how to build a streaming SQL applications.  Over the coming months, these tutorials will address the different components of streaming data applications, and provide worked examples and guidance.

Streaming Visualization, Part 1: Setting up

We’ll begin the series by looking at a typical streaming use case – displaying real-time… Read more →

The 18th World Congress on Intelligent Transport Systems (ITS) is being held in Orlando from October 16th – 20th, 2011. This is the leading event for intelligent transportation solutions, and attracts a large audience of government, technology and industry professionals. The event seeks to demonstrate advances in the application of new technology and smart transportation. Major areas of focus include the reduction of traffic congestion and improvement in  personal mobility.

With 800 million vehicles on the world’s roads today, a number forecast to grow to between 2 and 4 billion by 2050, it is clear… Read more →

I am going to discuss a SQLstream application for monitoring traffic flow in real-time. In this application, vehicles with GPS enabled devices transmit vehicle position along with other vehicle information such as speed and engine state. SQLstream receives this information as a real-time data stream and uses streaming SQL analytics to detect and predict the rapid onset of congestion on the road network in real-time.

Streaming SQL for Congestion Detection
The SQLstream application for congestion detection uses a typical streaming SQL processing pipeline. In this case, data is fed into the SQLstream pipeline using our Log File Adapter. SQLstream adapters provide an… Read more →

At SQLstream we have a comprehensive implementation of standard SQL windowing operations such as SUM, COUNT and AVG. Recently though we needed a more sophisticated function for a decaying weighted average that would emphasize more recent samples over older samples. We implemented a new EXP_AVG() operation, as shown in this example query:

SELECT rowtime, ticker, price,
+++++++++++++ORDER BY rowtime

EXP_AVG takes a value expression and an interval constant half life. In this example, two samples within the WINDOW are separated by 10 seconds, the older one will be… Read more →

August 23, 2011 by in Stream Processing & SQL

SQLstream is helping to predict earthquakes across the world in real time. The system has been developed by a consortium of universities and government agencies, with funding from NSF (National Science Foundation), to provide an infrastructure of networked tools for research in ocean science – constructing an internet-based system to collect and share data.

This is a large system with 16,000 land and sea-based sensors, each sending several channels of seismic event data at a rate of 40 data points per second. The application executes in an Amazon EC2 Cloud on a cluster of… Read more →

Mozilla Firefox Real-Time Download Monitor - Streaming Analytics on Hadoop

Since Tuesday’s announcement that the Firefox Download Monitor is powered by SQLstream, we’ve received a number of questions about how it all fits together. SQLstream server executes SQL statements, just like standard SQL, except the SQLstream’s queries run continuously, analyzing input data in real-time as it arrives. Statements are presented via JDBC or user friendly tools which use JDBC internally. Statements are compiled/prepared, the planner/optimizer chooses an access plan, and a runtime engine executes the plan. SQLstream is compliant with SQL2011 with just a couple of extensions. One extension includes the keyword STREAM as part of a SELECT… Read more →

March 23, 2011 by in Stream Processing & SQL

SQLstream has been powering Mozilla’s Firefox Download Monitor since 2009. A SQLstream based application has been continually aggregating hundreds of millions of download events, receiving minute by minute aggregations via a continuously running SQL SELECT statement using the SQLstream JDBC driver. A continuously running SELECT statement is syntactically and semantically identical to other SELECT statements with the addition that end of data is never returned in SQLSTATE by the FETCH associated with the cursor.

For the launch of Firefox 4, Mozilla again turned to SQLstream to enhance… Read more →

With service and sensor data growing at 60% CAGR, having both the raw power and correct architecture for processing streaming data is essential. IDC released recently estimates for the size of the ‘Digital Universe’ – a term used to describe every electronically stored piece of data. According to IDC, stored data will reach 1.8 million petabytes (1800 exabytes) by the end of 2011.

As a recent article in the Economist points out, all of this data raises significant processing performance and storage issues. Conventional database technology requires data to be stored, cleaned and aggregated… Read more →

February 21, 2011 by in Hadoop & Big Data Storage

Attendance at the PostgreSQL West 2010 Conference was encouraging considering a million people had gathered in the city to celebrate the World Series victory of the San Francisco Giants.

I’ve posted the presentation from the event (previously blogged about here).

We presented the concepts of Streaming GIS, integrating SQLsteam’s real-time streaming data analytics with the PostgreSQL-based Geographic Information Systems (GIS) engine PostGIS. With examples from SQLstream’s commercial traffic congestion monitoring application, we discussed how sophisticated high performance real-time geospatial applications can be delivered… Read more →

SQLstream’s founding engineer Sunil Mujumdar is set to present at PostgreSQL Conference: West 2010.

In a talk entitled ‘Streaming GIS using PostGIS and SQLstream’, Sunil will describe the SQLstream stream computing platform based on industry-standard SQL, and its integration with the PostgreSQL-based Geographic Information Systems (GIS) engine PostGIS.

Wireless sensors and internet services are generating data faster than conventional database technologies can process that data. In particular, mobile resource management requires the stream processing of high volume, location-based data. Solutions require new methods such as Streaming SQL to address these new sources of big data, in real-time. We’ll… Read more →

A streaming SQL query is a continuous, standing query that executes over streaming data. Data streams are processed using familiar SQL relational operators augmented to handle time sensitive data. Streaming queries are similar to database queries in how they analyze data; they differ by operating continuously on data as they arrive and by updating results in real-time.

Streaming SQL queries process dynamic, flowing data, in contrast to traditional RDBMSs, which process static, stored data with repeated single-shot queries. Streaming SQL is simple to configure using existing IT skills, dramatically reducing integration cost and complexity. Combining the intuitive power of SQL with… Read more →

October 26, 2010 by in Stream Processing & SQL

In the game industry, complex game logic needs to be applied to streams of events generated by gameplay.  In single player games, this logic is simply handled by applying the correct computations.  However, in an Internet based social game where millions of players interact together online, the problem takes on an entirely different dimension.  Storing the game events on disk inside of a database becomes increasingly difficult as the rate of gameplay events increases.  Logic and computation must be applied to the data and a disparate set of data must be queried to correctly update the game state.

The solution can… Read more →

October 18, 2010 by in Stream Processing & SQL

Businesses need to respond faster than ever to customer information and demands, which are arriving in rapidly increasing volumes from ever more diverse and distributed systems. This need for real-time business models can not be addressed by traditional integration and business intelligence solutions because streaming analytics and related concepts are central to the solution. The real-time model means responding immediately to new information as it arrives and streaming analytics is at the core of these next generation IT systems.

Increasing the speed of business under these pressures of rapidly increasing data volume and more diverse data sources has been expensive and… Read more →

October 4, 2010 by in Stream Processing & SQL

Railroads have used track side readers to scan bar codes on the sides of freight cars since the 1970s. Such sensors provided real time tracking of goods as they made their way from the supplier to the delivery point. Retail businesses increased the use of RFID tags in the past 20 years to track goods through the manufacturing process. Since the Indian Ocean tsunami of December 2004 the public has become aware of deep water pressure sensors which sit on the ocean floor to detect tsunamis and are intended to generate warnings about potential disasters.

The cost of sensors has decreased… Read more →

Last week I was on a panel for “Big Data” at Structure2010 – a GigaOm event. As usual, it was very well run and there was a large throng of silicon valley luminaries ranging from entrepreneurs to venture capitalists scattered in with some large customers and users of technology. We clearly have moved on a long way from the days when I was told to change my slides and remove the cloud graphic and replace it with a box because “clouds are cloudy” (direct quotation from a tier one venture capitalist – I wish to protect his identity… Read more →

There is a lot of buzz these days about the challenge of “Big Data”.  I’ll be speaking on the subject at GigaOM’s Structure2010, on the “DEALING WITH THE DATA TSUNAMI: THE BIG DATA” panel. There are many dimensions to the challenges posed by “Big Data”, which I’ve presented here as five separate but related themes.

Speed of data arrival

The first theme is speed.  When a lot of data arrive fast, it is often overlooked that they arrive in raw form and need to be processed or cooked before they can be of any real value. The processing normally… Read more →

Last year has been an interesting experience as I participated in a number of customer “Proof Of Concept” projects for SQLstream. Developing these real-time, stream computing projects greatly increased my appreciation for the advantages of an open, extensible and standards-compliant middleware infrastructure.

For example, I needed to implement an “edge detection” mechanism for a POC project. My colleagues at SQLstream recommended using “Bollinger bands” for determining outliers. So, I browsed through the  wikipedia entry for Bollinger Bands to learn more. Bollinger bands are very similar to standard deviations or quartile deviations. A Standard deviation measures variability or dispersion in data… Read more →

Just back from the 2010 Intelligent Transportation Society of America’s Annual Meeting.  For those unfamiliar with intelligent transportation, I am not referring to the “shovel ready” projects that have been funded by President Obama as part of the economic stimulus package. These projects were designed to spend money and create jobs, thereby, stimulating the economy. Unlike the federal “shovel ready” projects, “network ready” intelligent transportation technologies and projects are rapidly being adopted and implemented by local and state departments of transportation that must still operate under fixed or reduced budgets. These local and state DOTs are using new technologies… Read more →

Gravity Bear and SQLstream today announced a partnership to bring cutting edge real-time analytic technologies to games for social networks. Established to create a new breed of social games, Gravity Bear is poised to create engaging, original content for social gaming platforms. The relationship marks a significant point of convergence for the interactive entertainment and the enterprise software industries.

SQLstream is the first company to provide real-time monitoring and business intelligence using the ISO standard SQL language. Bridging the gap between operational intelligence systems and data warehouses, SQLstream greatly reduces the time it takes for information to flow between products and… Read more →

Contact Us

Would you like to know more about stream processing?

+1 877 571 5775

Ask a Question