Real-time Big Data means the streaming, continuous integration of high volume, high velocity data from all sources to all destinations,  coupled with powerful in-memory analytics. It’s a paradigm shift from conventional store and process systems that’s playing well here at the Intelligent Transportation Systems and Solutions World Congress here in Vienna.

(more…)

We’re at the Intelligent Transportation Society’s World Congress in Vienna, Austria next week, Mon 22 – Fri 26 October. The theme for this year is ‘Smarter on the Way’ and this is the first year since 2009 that it’s been held in Europe. The full panoply of intelligent transportation issues is addressed, everything to do with software, hardware and firmware that improves the traveler experience, transportation network utilization and capacity optimization. However, the combination of real-time traveler information systems, wireless sensors, GPS and predictive analytics is a specific focus. And increasingly, telematics and V2I/V2V applications are represented.

We’re going to be exhibiting, but also presenting on a special interest panel session: Optimizing ITS from a Customer Service Perspective. Essentially looking at how new technology can be used to make a significant and noticeable difference to a traveler’s experience. Our focus is to bring to bear our experience of consumer and customer quality of experience monitoring from other industries, particularly telecommunications. Social media and Twitter also has a role to play, with significant interest is extracting semantic information from Tweets to detect serious traffic incidents and congestion as it happens. However, there are issues with relying on social media, as I’ve discussed here in a recent ITS International article, Social media mooted for traffic management.

Industries such as transportation, M2M and telematics are clearly leading the way in deploying real-time Big Data applications. And that’s understandable, it’s sensor networks and the Internet of Things that’s creating vast volumes of high velocity data where each event or record has value for only a very short period of time after its creation. If you’d like to understand more about streaming geospatial analytics from live vehicle GPS data, the video below presents a brief overview of a recent deployment.

Continuous ETL for Google BigQuery

There is another significant difference from the traditional business intelligence and real-time analytics market . The mainstream Big Data market is focused entirely on Business Intelligence, a one way function for  collecting data in a static store, and displaying the results on a graph or report. Visualization is also an important consideration for industry, but streaming integration is the first requirement, and the ability of the platform to support real mission critical applications a close second.

The Intelligent Transportation industry has all three requirements in abundance – vast volumes of fixed, wireless GPS sensor data that must be integrated seamlessly across the operational systems, mission critical operation as significant safety, environmental and traveller impact decisions are made as a result, and the abilty to visualize network flow, congestion and alarms.

So if you’re in Vienna, Austria next week, drop by our booth, P21.

Event report from ITS California,  Annual General Meeting in Long Beach, Nov 13 – 15, 2011.

ITS California’s AGM is SQLstream’s local intelligent transportation event, and an opportunity for public and private sector companies to discuss those issues specific to the state of California. The event has grown in size significantly over the past few years, with new organizations attending, all keen to contribute and discuss different perspectives on the various transportation problems. ITS-CA was established in 1994 as a not-for-profit organization with the remit to foster the adoption of ITS technology across the state. Its funding is a mix of public and private industry.

California has some acute transportation issues. For example, in Los Angeles County, the number of new residents is set to increase traffic on already congested roadways by an estimated 39 percent. Roadway expansion in this period is set to increase by only 3 percent, resulting in congestion levels rising by more than 200 percent in the next 25 years. Obviously addressing these issues ha been the focus of the Californian transportation industry as a whole, but ITS-CA serves as a focus point once per year to discuss in a wider forum.

The theme of this year’s event therefore (not unsurprisingly) was “Discovering Keys to the Next Decade”, with session focused on:

• Support for connected vehicle deployments.
• Transportation systems efficiency, for example reliable travel time and pricing.
• Analyzing the success of traffic light systems and how close we are to the ‘continuous green’ vision.
• Improving traveller information systems (of specific interest to SQLstream).
• The effectiveness of schemes such as car sharing.

Although there’s always an over-arching theme or direction that emerges, and in this case it was the increasing presence of the private sector, and how private sector innovation is helping to drive the adoption of new technology and approaches. This was a common topic at the ITC-CA in Berkley last year, and has continued to grow. It was also a trend that was raised frequently at the recent ITS World Congress in Orlando. Long may it continue.

It struck me that the underlying theme of the conference could be described as ‘low risk innovation’.  An oxymoron?  At first glance, yes, but in this case it describes the cautious adoption of new technology while protecting existing investment.

The perception of the ITS industry, rightly or wrongly, has been one of an industry focussed on major manufacturing and hardware deployment projects, rather than software and new technologies. Therefore perhaps the industry hasn’t seen the level of growth and innovation that has occurred in other areas, telecommunications being a good example.  Or perhaps it simply hasn’t been possible until new technologies come along with a sufficiently compelling business case – lower cost solutions and faster to deploy.

New technology introduction – complementary and overlaid

GPS and Bluetooth featured strongly at the event on both the exhibition floor and in the breakout panel sessions.  It was clear however that the preference was to use these either as a proof concept across a small area, or where monitoring infrastructure exists, as complementary solutions, extending the accuracy and scope of existing fixed-road sensor deployments.

Software solutions and architecture

It was interesting how many of our booth visitors were initially attracted by SQLstream ITS Insight, but who were actually looking for more of a horizontal platform solution.  Yes, out of the box capability was important to get solutions up and running quickly, but wider concerns touched on some fundamental software engineering principles – openness, scalability and interoperability.

Open and flexible

The focus on manufacturing and hardware oriented projects tends to produce software support systems that are built to do just that – support the particular hardware installed for that project.  This leads to capable and often feature-rich systems, but systems that are difficult to extend and configure for new technologies and requirements.  There was a definite theme in the questions being asked at our booth for openness – open platforms, where an agency’s IT department, or consulting partner, can add new applications easily, and in fact are encouraged to do so.

Performance and scalability

Systems are required to scale in two ways.  The first is raw performance as the number of sensor events increases and real-time performance is required. Secondly, and perhaps less obvious, is that transportation agencies are looking to consolidate systems and provide common systems across multiple counties and even at the state level.  This highlights immediately the scalability issues with existing systems.   As the user base increases, the geographical scope of the system increases, and the drive for real-time information increases, the weakness of existing systems have been exposed.

Standards

The emergence of IT standards is a sure sign of increasing maturity in any industry.  This represents a move away from limited, siloed solutions to consideration of the wider integration issues – integration with network hardware, but also the integration between management systems.

A final word …

One final word, not quite a trend yet, but ITS appears to be embracing the Cloud as a solutions platform, as a mechanism for providing access to applications, but also for scalability and as a lower cost solution where large infrastructure deployment would be required.

A defining feature of the show is the Technology Showcase featuring  demonstrations from some of the technologies and applications that are bringing the future of transportation to life.  Each ‘village’ covers a specific theme such as Safety, Mobility, Environment/Sustainability and Pricing.  Environment/Sustainability focuses on the potential for reducing emissions.  Interesting demonstration offered by Imperial College, London, that will use a pollution sensor mounted on the roof of a demonstration vehicle travelling around the site.  Data is transferred over a GSM connection and viewed over the internet. Another is Ricardo Engineering’s demonstration of improving fuel efficiency using GPS navigation and traffic signal phase and timing data.

On the theme from yesterday of using simulators to demonstrate the benefits of technology, Toyota’s Star safety system is a star of the show – attendees get a spin in a off-road driving simulator, once with all electronic aids turned off, and once with traction control, stability control, brake assist and anti-lock braking turned on.

And for SQLstream’s day, another busy day with many questions and demonstrations of real-time traffic analytics and congestion detection with SQLstream ITS Insight.  Our core Stream-to-Business platform, and the ability to integrate and extend SQLstream ITS Insight easy and quickly, is generating significant interest.

The ITS World Congress claims to be the largest transportation event in 2011. Certainly the range of attendees and exhibitors is impressive, from software products to the latest in roadside hardware infrastructure. Some interesting themes emerging. Vehicle to vehicle communication is generating a lot of interest, and electronic driver aids. Some great simulators as well to amuse the attendees.

‘Real-time’ is also a common theme across the exhibition hall. And in particular the use of new technologies such as Bluetooth and GPS. Damian Black, SQLstream’s CEO, was speaking in a session today on how to achieve accurate arterial travel time. This is a hot topic right now, where existing in-road sensors are too expensive and the solutions too inaccurate. Bluetooth and GPS are the two emerging, although not necessarily competing, technologies – both can be used simulataneously to reinforce the other.

SQLstream in action – Day 1

It was great to have so much interest at the booth for our real-time intelligent transportation solutions. We announced the public launch of our SQLstream ITS Insight solution this morning. Perhaps we’re one of the few new companies at the show, but the ability to build real-time traffic analytics solutions based on maximizing the use of all available sensor data is generating a lot of interest.

SAN FRANCISCO, CA, October 17, 2011 - SQLstream Inc. today announced the public availability of SQLstream ITS Insight, the first real-time solution for reducing congestion to exploit low cost wireless GPS data as a complement to existing fixed-road sensor investment. Transportation Agencies are already benefiting already from SQLstream ITS Insight, using it to deliver real-time Travel Time and congestion detection solutions. The official public launch for SQLstream ITS Insight is today at the ITS World Congress, Orlando.

For commuters in their cars, traffic lines are lengthening and travel times are increasingly unreliable. Congestion is growing globally and transportation agencies are already struggling with their existing congestion management systems. Traditional approaches are expensive to install and maintain, and they provide very limited information and network coverage.

“Transportation Agencies are revolutionizing their approach to transport network management”, said Damian Black, SQLstream CEO. “SQLstream is delighted to be the core of their strategy for a single, real-time Intelligent Transportation platform.”

About SQLstream ITS Insight

SQLstream’s ‘Insight’ product range offers fast start solution packs for industry markets based on SQLstream’s core Stream-to-Business platform. SQLstream ITS Insight is a real-time traffic analytics and management platform for Intelligent Transportation agencies, offering real-time Travel Time, and sophisticated congestion detection algorithms that combine real-time and historical trend data. With SQLstream ITS Insight, transportation agencies will:

• Significantly reduce the cost of fulfilling congestion reduction targets
• Implement travel time improvements in just weeks rather than years
• Achieve the impossible – effective real-time insight for arterial routes

SQLstream will be presenting live demonstrations of SQLstream ITS Insight at ITS World Congress, Orlando, booth #1366.

About SQLstream Inc.

SQLstream’s Stream-to-Business platform analyzes real-time service and sensor data streams to deliver instant alerts, analytics and immediate answers to business decision makers. Using the industry standard SQL language, SQLstream executes queries on the wire, before data reaches the warehouse, enabling businesses to make smarter decisions sooner. SQLstream adds real-time operational intelligence, monitoring and control to existing systems while reducing total cost and complexity. SQLstream is headquartered in San Francisco, California and is on the web at http://www.sqlstream.com. For further information, please call Ronnie Beggs (877) 571-5775, or email pr@sqlstream.com.

Visit SQLstream on Booth #1366

Technology and innovation are central themes of this year’s ITS World Congress.  There’s been much written about the issues of congestion, green transportation schemes and improving personal mobility, not least in this blog.  At SQLstream we’ve been playing our part to help revolutionize the Intelligent Transportation industry.  It’s clear that the concepts of streaming data and real-time analytics are entering the main stream – from low level Big Data toolkits that require a streaming, low latency front end, to the real world of sensor networks and industries such as smart grid and telecommunications.

This is just as true in transportation.  Here we have an industry with vast volumes of sensor data, a need for sophisticated real-time analytics, and platforms capable of driving real-time process automation.  We’ve been working with a number of transportation agencies for some time, and are about to launch a new ‘Insight’ product for intelligent transportation.  Our ‘Insight’ range provides tools and out of the box support for specific industry verticals based on our core Stream-to-Business platform.

Google Earth Display for Road Traffic Congestion

For Intelligent Transportation this means processing sensor data from GPS and fixed-road sensors, to deliver applications such as real-time Travel Time, live congestion detection and network KPI reporting.

Should you be attending the ITS World Congress, we’d be delighted to see you on our booth (#1366) for a demonstration.

The 18th World Congress on Intelligent Transport Systems (ITS) is being held in Orlando from October 16th – 20th, 2011. This is the leading event for intelligent transportation solutions, and attracts a large audience of government, technology and industry professionals. The event seeks to demonstrate advances in the application of new technology and smart transportation. Major areas of focus include the reduction of traffic congestion and improvement in  personal mobility.

With 800 million vehicles on the world’s roads today, a number forecast to grow to between 2 and 4 billion by 2050, it is clear that transportation management  systems will need to analyze real-time sensor and GPS data dynamically on a massive scale to reduce congestion and optimize personal mobility. The objective is to achieve a fluid and reliable transportation network, that can respond dynamically to changing loads and conditions, and provide consistent and acceptable travel times.

The performance of a transportation network can be measured based on road usage (number of vehicles), and the travel speed and time from origin to destination.  Today’s traffic management systems rely on historical analysis of data from fixed sensors.  However, roadside and in-road sensor projects are very expensive to install and maintain. As a consequence, only a very limited view of the overall road network is available,  with sensor deployments focusing on primary routes and major intersections only. Also, fixed sensors tend to report traffic flow – at best a secondary measure of the real requirement –  congestion.

Most important however is the lack of real-time, dynamic behaviour from existing traffic management systems.  Flow control, for example at intersections and on freeways, is activated at specific times based on the historical analysis of the fixed sensor data – this helps, but is unable to react to changing patterns of traffic flow and congestion.

One approach to the problem is to introduce the latest wireless GPS sensor technology.  Wireless GPS sensors have two significant advantages:

1. Immediate and real-time information on vehicle speed and location.
2. Low cost solutions that can be deployed quickly, with little or no maintenance.
3. Provides a direct measure of vehicle speed and the ability for real-time and accurate measure of congestion.
4. Complete network insight – highways and arterial routes – at the granularity of a few meters.

For example, when one national road agency was re-evaluating its approach to intelligent transportation systems, it identified wireless GPS technology as both a significantly cheaper and potentially much superior solution to congestion detection and Travel Time. SQLstream was selected as the real-time traffic analytics and congestion detection platform based on processing in-vehicle GPS sensor data. The SQLstream solution enabled the agency to cancel a $20million fixed sensor program,  and to build a real-time traffic management platform based on SQLstream’s ITS Insight.

We will be demonstrating our real-time traffic management capabilities on our stand at ITS World Congress in Orlando.  In addition, our CEO, Damian Black will be participating in a number of related panel sessions on arterial travel time solutions and real-time data management for intelligent transportation.  For those attending ITS World Congress, please visit us for a demo at Booth #1366, or visit our website for more information on SQLstream and real-time transportation management systems.  We look forward to seeing some of you at least some at the show.

I am going to discuss a SQLstream application for monitoring traffic flow in real-time. In this application, vehicles with GPS enabled devices transmit vehicle position along with other vehicle information such as speed and engine state. SQLstream receives this information as a real-time data stream and uses streaming SQL analytics to detect and predict the rapid onset of congestion on the road network in real-time.

Streaming SQL for Congestion Detection
The SQLstream application for congestion detection uses a typical streaming SQL processing pipeline. In this case, data is fed into the SQLstream pipeline using our Log File Adapter. SQLstream adapters provide an interface to sources and targets such as databases, log files, network sockets and mail servers. Adapters are built using SQL/MED specification which is part of ANSI SQL standard. In this application, each log file contains the vehicle positions on the road network for the latest minute.

The conditioning pipeline performs data cleansing operations such as rejecting poor quality data (records with missing or out-of-bounds columns) followed by mapping of vehicle positions (lat/long pair) to a “road element” of the road network using a UDX to perform geo-spatial lookups in an external road network database.

The diagram and the example SQL below show our implementation of a streaming SQL pipeline for congestion detection. Each vehicle reports its position and speed every minute. Two consecutive vehicle positions are then used to interpolate vehicle speeds for each road element on the vehicle path between reporting positions. The interpolated speed is based on actual distance traveled by the vehicle between two consecutive reports. The interpolated speed is calculated in a User Defined Transform(UDX). The UDX is written in Java. The UDX also associates a confidence factor with each interpolated speed value based on the position of the road element relative to endpoints of the vehicle path.

Streaming Traffic Flow Analytics
As illustrated below, the analytics pipeline calculates 15, 5, 4, 3, 2 & 1 minute moving average speeds for each road element. Each road element is color coded based on the 15-minute moving average speed. The results are streamed to a Google Earth display.

Detecting the rapid onset of congestion
Congestion is detected by comparing moving averages for the larger time window with that for the smaller time window. For example, comparing a 2-minute average with a 1-minute average:

Note that these estimated speeds are over overlapping windows and as such slowdown thresholds are set accordingly.

Fine tuning slowdown thresholds and other information, such as the proximity of traffic lights and the volume of vehicle reports in each time window, improves the quality of congestion detection algorithm.

The Google Earth screenshot illustrates real-time traffic view as well as detected slowdowns as pins. The severity of the slowdown is indicated by different shades of red.

Attendance at the PostgreSQL West 2010 Conference was encouraging considering a million people had gathered in the city to celebrate the World Series victory of the San Francisco Giants.

I’ve posted the presentation from the event (previously blogged about here).

We presented the concepts of Streaming GIS, integrating SQLsteam’s real-time streaming data analytics with the PostgreSQL-based Geographic Information Systems (GIS) engine PostGIS. With examples from SQLstream’s commercial traffic congestion monitoring application, we discussed how sophisticated high performance real-time geospatial applications can be delivered quickly and easily using standards-based SQL.

Streaming GIS using PostGIS and SQLstream at PostgreSQL West 2010

Just back from the 2010 Intelligent Transportation Society of America’s Annual Meeting.  For those unfamiliar with intelligent transportation, I am not referring to the “shovel ready” projects that have been funded by President Obama as part of the economic stimulus package. These projects were designed to spend money and create jobs, thereby, stimulating the economy. Unlike the federal “shovel ready” projects, “network ready” intelligent transportation technologies and projects are rapidly being adopted and implemented by local and state departments of transportation that must still operate under fixed or reduced budgets. These local and state DOTs are using new technologies to “Do More with Less.”

Intelligent Transportation aims to reduce costs, delays, pollution, injuries and deaths by connecting infrastructure control and monitoring systems to the network and enabling these systems, and their operators, to communicate in real-time. Some examples of intelligent transportation solutions and control systems include dynamic speed limits that change according to traffic and road conditions, stop lights that know when you can go and the FasTrak electronic toll system that reduces congestion on the Golden Gate Bridge and other Bay Area bridges. Real-time technology is essential if these dynamic control systems are to collect your toll at 45 miles per hour or detect when it is safe to proceed through an intersection.

All of these intelligent transportation systems and devices can be thought of as “sensors” on the network. The data is collected by the sensors, streamed to a server, analyzed and eventually stored in a warehouse. (Imagine the final scene from Raiders of The Lost Ark, except with crates full of hard drives). Meanwhile, the analytic results are communicated back to the original sources (stop lights, toll booths and electronic road information signs) as well as to the mobile devices in your vehicle.

In some cases, new intelligent transportation solutions need to be integrated with legacy systems. In other cases, they simply need to be able to talk to each other. Thus, it becomes imperative that all new intelligent transportation solutions be built on a set of common, open standards. In the long run, solutions built on open standards reduce the total costs to those who implement and maintain the solutions. Open standards, and in particular, the global use of open data standards, within the intelligent transportation industry is essential, not just so that different sensors on the network and IT solutions can communicate with each other, but so that drivers can experience consistent and safe journeys as they cross from federal highways to state and local roads, always in contact with intelligent transportation systems that control these roads.