“The Total Cost of Performance in a Massively Connected World” Session to be Presented at GigaOM Structure on June 19^th at 4:55PM PST in San Francisco

San Francisco, CA | June 18, 2013 –

SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today announced that SQLstream CEO Damian Black has been invited to speak about the cost of achieving real-time performance from Big Data technologies at GigaOM’s Structure 2013, June 19-20.

Damian will be exploring the cost and technical limitations of generating real-time operational intelligence from different Big Data technologies. As data-management architectures evolve, each new technology looks to increase real-time visibility and reduce the latency-to-action and decision making. Yet technologies such as Hadoop have an associated cost and a practical performance limit beyond which real-time, high velocity distributed data management becomes unfeasible.

Held at Mission Bay Conference Center at UCSF, and now in its sixth year, GigaOM Structure gathers data scientists, business leaders and investors interested to learn more about the emerging business opportunities for Cloud and Big Data. Structure 2013 will focus on how real-time business needs are shaping IT architectures, hyper-distributed infrastructure and creating a cloud that will look completely different from everything that’s come before. Damian will present The Total Cost of Performance in a Massively Connected World at 4:55PM PST on Wednesday, June 19.

Damian Black, was selected for his pioneering work in real-time and Big Data software and his commitment to reducing latency-to-action for business operations.  Black has participated in many GigaOM events, including the first ever Big Data panel session at GigaOM Structure 2008. “GigaOM has its finger on the pulse yet again with the focus on addressing real-time business needs from Big Data,” said Damian Black, SQLstream CEO. “The drive towards high velocity data coupled with low latency actions marks a new frontier for Big Data, one where the cost of performance is driving streaming data technology to the fore.”

If you are a member of the press or analyst community and are interested in setting up a meeting with SQLstream at GigaOM Structure:Data, please contact Ronnie Beggs via email at ronnie.beggs@sqlstream.com.

About SQLstream

SQLstream (www.sqlstream.com) is the pioneer and innovator of a patented Streaming Big Data Engine that unlocks the real-time value of high-velocity unstructured machine data. SQLstream’s s-Streaming products put “Big Data on Tap™ – enabling businesses to harness action-oriented analytics, with on-the-fly visualization and streaming operational intelligence from their log file, sensor, network and device data. SQLstream’s core data streaming technology is a massively scalable, distributed platform for analyzing unstructured Big Data streams using continuous SQL, with support for streaming SQL query execution on Hadoop and other enterprise database, data warehouse and data management platforms. SQLstream’s headquarters are in San Francisco, CA.

Media Contact

Ronnie Beggs, VP Marketing

Company Name |  SQLstream, Inc.
Contact E-mail | pr@sqlstream.com
Website URL | www.sqlstream.com

Resources

GigaOM Structure agenda

GigaOM Structure registration

SQLstream GigaOM Structure:Data 2013 Damian Black interview

Ronnie Beggs Participating on “Big Data: Architecting Systems at Speed ” Panel at E2 Conference, June 18^th at 2:30PM ET, Boston

San Francisco, CA | June 17, 2013 – SQLstream Inc., the streaming Big Data platform, today announced that SQLstream has been invited to speak at E2 Conference on the architectural challenges of generating real-time operational intelligence from high-velocity Big Data streams.

The panel gathers industry experts to explore how real-time, high velocity Big Data streams can be processed rapidly in order to make meaningful decisions quickly.  It has been assumed that volume is the fundamental problem presented by Big Data.  It’s not. It’s data velocity and speed to action. The panel lead by esteemed analyst Johna Till Johnson of Nemertes Research will analyze the challenges of rethinking how Big Data processing is architected.

Held at Boston Marriott Copley Place, E2 is the only conference focused entirely on the choices and challenges organizations face as a result of shifts in technology. A new event in the Big Data world, E2 offers decision makers in IT, data scientists, and business leaders a progressive forum where they can engage in sessions on the future of software, and explore how it can dictate the needed changes in corporate strategies.

“What I like about E2 Conference is its focus on the challenges that organizations face as a result of technology shifts, rather than the technology itself,” said Ronnie Beggs, SQLstream’s Vice President Marketing.  “Our panel is timely as well, given the wider focus on the cost of Big Data performance for low latency actions from high velocity machine data.”

If you are a member of the press or analyst community and are interested in setting up a meeting with SQLstream at GigaOM Structure:Data, please contact Ronnie Beggs via email at ronnie.beggs@sqlstream.com.

About SQLstream

SQLstream (www.sqlstream.com) is the pioneer and innovator of a patented Streaming Big Data Engine that unlocks the real-time value of high-velocity unstructured machine data. SQLstream’s s-Streaming products put “Big Data on Tap™ – enabling businesses to harness action-oriented analytics, with on-the-fly visualization and streaming operational intelligence from their log file, sensor, network and device data. SQLstream’s core data streaming technology is a massively scalable, distributed platform for analyzing unstructured Big Data streams using continuous SQL, with support for streaming SQL query execution on Hadoop and other enterprise database, data warehouse and data management platforms. SQLstream’s headquarters are in San Francisco, CA.

Media Contact

Ronnie Beggs, VP Marketing, Company Name |  SQLstream, Inc.

Phone: +1 415 758 8342
Contact E-mail | pr@sqlstream.com
Website URL | www.sqlstream.com

Resources

E2 agenda

E2 registration

Sensors Expo is the leading industry event for the types of intelligent sensor-integrated systems that are driving the next generation Internet of Everything, Industrial Internet, telematics and Machine-to-Machine services. As a key sponsor in the Big Data and Wireless Systems pavilion, and speaking in the Big Data track, we found the event a great litmus test for the current state of the sensor-related systems’ industries.

This year, the focus was very much on the monetization of intelligent sensors networks. This included how businesses and consumers can benefit from the Internet of Things, but also a focus on infrastructure costs, in particular, architectures for managing the high cost of wireless backhaul for large scale sensor data transmission.

The clues were there at last year’s event, where the focus was not so much real-time operational intelligence and monetization, rather the more fundamental issue of real-time streaming sensor data integration. The main question twelve months’ ago was how can such huge volumes of data from many different sources and locations be integrated into the operational platforms, and at an affordable cost.

Drinking from the Sensor Firehose

The Internet of Everything is the new frontier for real-time Big Data management, where data velocity now trumps data volume as the primary driver, where the geographical distribution of streaming data adds new levels of complexity, and yet the useful lifetime of data, the window within which to make a decision, has decreased dramatically. As data management architectures evolve, each new technology looks to increase real-time visibility and reduce the latency to actions and decision-making.

Which is exactly problem that SQLstream’s VP of Americas, Glen Hout, presented in his session on Real-Time and Big Data: The Perfect Union for Large-Scale Sensor Network Management. SQLstream is a streaming Big Data management platform, sensor data is processed in real-time as they are created, and answers generated with sub-second latency. SQLstream also collects unstructured sensor data of any type and from any number of sources simultaneously, and can deliver multiple aggregated and real-time intelligence to external systems such as operational management platforms, existing data warehouses and Big Data storage platforms.

The Intelligent Edge

However, the issue with the cost of transmission backhaul is also addressed by SQLstream. SQLstream’s agent framework enables intelligent data collection agents to be deployed as close to the data source as possible. Intelligence Agents offer filtering, aggregation and analytics capability at the edge of the network. Depending on the information content in the data, the volume and therefore cost of real-time transmission will be reduced significantly.

So the future is bright, the future is real-time operational intelligence. And to achieve that, requires the next generation of streaming data management and true real-time operational intelligence platforms.

SQLstream was named as one of the top 100 companies that provide technologies and services for data management by the Database Trends and Applications (DBTA) magazine.

A result of lengthy research and compiled by the publication’s editorial staff, the first “DBTA 100: The Companies That Matter Most in Data” list recognizes industry leaders that are helping organizations derive tangible value and insight from Big Data.

Read more, HERE.

InfoArmor and SQLstream Team Up to Deliver the Next Generation of Continuous Identity  Monitoring Services at Massive Scale 

SAN FRANCISCO – JUNE 5^TH, 2013 – SQLstream, Inc., the Streaming Big Data Company, today announced that InfoArmor, a U.S. leader in identity theft monitoring and internet surveillance, selected SQLstream as the real-time engine for the next generation of its identity monitoring and surveillance platform. SQLstream’s s-Streaming products enable InfoArmor to meet the rapid growth in demand for its real-time identity monitoring services. SQLstream was chosen for its capacity to capture, process and integrate in real-time high volumes of unstructured data coming from a large variety of sources, its massive scalability for high velocity real-time operational intelligence, and its simple, fast deployment.

Identity theft in the U.S. alone claimed 12.6 million victims in 2012, a rise of over 1 million from 2011, resulting in more than $21 billion in damages. InfoArmor’s identity protection services are helping detect and correct identity theft before any damage is done or costs incurred. Real-time identity theft monitoring has complex, rules-based alerting intelligence that must be applied over a large number of different data sources simultaneously.

 “We needed a real-time operational intelligence platform that could scale to new levels of data acquisition, conditioning, analytics and alert delivery,” said Christian Lees, CTO of InfoArmor. “We evaluated building our own and explored other vendors, but chose SQLstream because they met our requirements entirely and they provided the only 100% ISO ANSI/SQL standards-based streaming platform. That enabled us massive scalability, a very fast deployment and a highly competitive TCO.”

The system captures and parses multiple data feeds in different XML-based formats. SQLstream parses and conditions the data feeds on the fly, and applies a sophisticated rules catalog across all data, delivering real-time alerts to consumers through SMS when rules and specific combinations of rules are breached.

“We’re delighted to have been chosen by InfoArmor for such a challenging operational Big Data requirement,” said Damian Black, CEO of SQLstream. “Seeing SQLstream making a real difference to people’s lives is a validation of our approach to operational intelligence and predictive analytics.”

Learn more about SQLstream’s security solutions today.

About InfoArmor

InfoArmor is a leader in offering identity theft protection and privacy management solutions to businesses. By focusing on employee benefits, InfoArmor provides a turn-key, value-adding benefit with easy employer administration. Rather than solely focusing on credit fraud, InfoArmor uses industrial strength technologies to enable consumers and businesses alike to monitor and identify fraudulent activity. For more information, visit http://www.infoarmor.com.

San Francisco, CA | May 30, 2013 – SQLstream Inc., the Streaming Big Data Company, announced today that SQLstream VP Americas, Glenn Hout, has been invited to speak on real-time operational intelligence and prescriptive analytics for the Internet of Everything at Sensors Expo 2013, Jun 4-6.

Held in Rosemont, Illinois, and now in its 27th year, Sensors Expo is the leading industry event in its class and gathers the world’s top engineers, scientists and business leaders involved in the development and deployment of sensor networks and systems. Areas of focus include sensor Big Data analytics, wireless sensor networks for Machine-to-Machine services, telematics and smart energy.

Glenn Hout is presenting “Real-Time and Big Data: the Perfect Union for Large-Scale Sensor Network Management” at 10:10AM CT on Thursday, June 6^th.  SQLstream was selected to address the Sensor Expo audience in the Big Data and Analytics track based on its pioneering work in real-time operational intelligence and streaming integration of prescriptive analytics for industrial sensor, telecommunications, intelligent transportation, telematics and M2M networks and services.

“It’s always a pleasure to engage with an audience so familiar with the practical implications of Big Data,” said Glenn Hout, SQLstream VP Americas. “The Internet of Everything is the new frontier for Big Data, and SQLstream’s prescriptive analytics engine is the only platform able to meet its real-time requirements. Scaling up to handle sensor data velocities of millions of records per second with ultra-low latency response is the prerequisite for generating and acting on operational intelligence.”

If you are a member of the press or analyst community and are interested in setting up a meeting with SQLstream at Sensors Expo, please contact Ronnie Beggs via email at ronnie.beggs@sqlstream.com.

The Internet of Everything is the new frontier for real-time and Big Data, where Velocity now trumps Volume as the primary driver, where the geographical distribution of streaming data adds new levels of complexity, and yet the useful lifetime of data, the window within which to make a decision, has decreased dramatically.

Industries such as telecommunications, telematics, M2M and the Industrial Internet are starting to generate high velocity data streams at rates of millions of records per second. Gaining actionable insight from data of this magnitude and speed may be technically feasible to a point for the elastic scale-out architectures of the leading Big Data and Cloud technologies, but at what point does it cease to be financially viable?

The Technology Tipping Point for Real-time Operational Intelligence

As data management architectures evolve, each new technology looks to increase real-time visibility and reduce the latency to action and decision-making. Yet each technology has an associated cost and a practical limit beyond which real-time, distributed data management becomes commercially infeasible and then technically impossible.

Big Data technologies emerged due to the combination of lower cost commodity hardware and Cloud infrastructure, coupled with the cost and the technical challenges of processing massive volumes of unstructured machine data (server logs, sensors, IP network data for example) rapidly using conventional “store-first, query second” relational database technology. This lead a charge towards new elastic scale out processing frameworks based on HDFS and Map-Reduce. However, as Hadoop-based and other similar technologies mature, the true cost of real-time and Big Data for Hadoop is starting to emerge.

For low latency operational intelligence applications, Big Data storage-based solutions are proving to be costly and complex to deploy, and technically challenging to scale for high velocity / low latency performance. Furthermore, today’s Cloud multi-tenant and shared infrastructures introduce unacceptable latency to real-time business analytics, and the cost ramps up significantly as ingress and egress data velocities increase.

The ROI and technical tipping points for storage-based (Big Data and traditional RDBMS-based architectures) is very much at the low end of the scale, typically in the order of 10,000 records per second. This is sufficient to process a Twitter firehose at full speed, but falls significantly short for industry applications for real-time operational intelligence. For example:

• Operational performance & QoS monitoring in a 4G cellular network generates cell and call detail information at many 10s of millions of events per second.
• Telematics applications are generating data rates of between 5 million and 20 million records per second for a small installation.
• For IP service networks, IP probe data regularly exceeds data rates of 10 million records per second.

These data rates are well within the technical capabilities of SQLstream’s Big Data streaming platform, and importantly, SQLstream has a proven ROI for high velocity data applications. That’s not to say in-stream analytics is the only platform required, and in fact SQLstream deploys Hadoop HBase as its default stream persistence platform, and a typical architecture deploys SQLstream for the in-stream analytics with continuous feeds of raw and operational intelligence from SQLstream to existing operational control and data warehouse platforms. The resulting architecture offers an attractive overall ROI and eliminates the technical tipping points for low latency, high velocity data management.

More on the cost of real-time performance in a Big Data world to follow as part of this series in subsequent blogs.

SQLstream sponsored the recent IE Group Big Data Innovation Summit in San Francisco where I also presented on streaming SQL for Hadoop, and extending Hadoop for real-time operational intelligence. As Big Data technologies and Hadoop push further into mainstream enterprises, so the need for real-time business operations is an important parallel trend. ‘Real-time’ and ‘Hadoop’ had been considered synonymous by some, yet surprisingly, people are surprised when Hadoop does not seem to be as real-time as they hoped. This should not come as a surprise, as Hadoop as many strengths, but was never intended for low latency, real-time analytics over high velocity data.

Click to View Damian’s Presentation on Slideshare

Real-time Big Data or real-time Big Data?

Which raises the question, what do we mean by real-time? Many products have emerging that claim ‘real-time’ analytics over Hadoop. Yet Hadoop remains a batch processing framework, and struggles to deliver low latency analytics against high velocity streaming data, struggling due to the same limitations as existing RDBMS-based data management platforms. These ‘real-time’ products may generate rapid results over the stored data, but ignore the latency introduced by data collection and storage, and also ignore the resource load of repeated execution of queries to process newly arriving data. The latency issue may not be apparent for slower data streams, such as twitter feeds for example, but with the data rates of machine data in the world of telecommunications, industrial automation, M2M and large scale security intelligence for example, the problem rapidly becomes extreme.

SQLstream’s core stream computing platform, s-Server, processes high velocity data as soon as they are generated, executing continuous SQL queries and analytics directly over log files, sensor feeds and any other machine-generated data source. We measure real-time form the time of data creation, eliminating completely the latency introduced by collecting, storing and the repeated updates of results.

Drive real-time actions with streaming operational intelligence

We discussed in a previous blog how real-time operational intelligence eliminates the chasm between business operations and analytics. Operational intelligence is about more than the collection and analysis of log file and machine-generated data. One of the advantages of stream computing is the ease with which predictive analytics can be applied over multiple data streams. This makes it possible to alert on time and space-based patterns of machine, user and consumer behavior that are predictors of some future event – a security breach, network failure or service fault.

And true operational intelligence platforms need to go one step further – true real-time platforms must do more than visualize results on a dashboard – it’s essential to connect back to application and operational systems, and to drive automated updates. Security breaches can be avoided, network resilience mechanisms activated, and service faults corrected before SLA breaches occur and customers are aware of the problem.

Real-time operational intelligence on Hadoop

So what does this mean for Hadoop? Streaming is not a new technology, but approaches streaming technologies have focussed on single source problems, and have been deployed as standalone platforms for low velocity use cases. With SQLstream, standard SQL queries, albeit continuously executing SQL queries, execute to join, group, partition and analyze real-time machine data streams. There is a further difference – SQLstream’s s-Server streaming SQL platform can also be deployed as a streaming SQL query extension for Hadoop.

A number of streaming Hadoop scenarios are supported:

• Stream persistence – Hadoop HBase as an active archive for streaming data and derived intelligence using the Flume API. SQLstream also performs continuous aggregation  to support high velocity streams without data loss.
• Stream replay – restream the complete history of persisted streams from HBase for ‘fast forwarding’ of time-based and spatial analytics. Various interfaces can be utilized, including Cloudera’s Impala.
• Streaming data queries, joining streaming real-time data with historical streams and intelligence persisted in HBase.

Making the Elephant fly

Accelerating Hadoop to process live, high velocity unstructured data streams delivers the low latency, streaming operational intelligence demanded by today’s real-time businesses. Hadoop has been the driving force behind Big Data Analytics but as the technology hits the mainstream, many industries are seeking to take a step further and eliminate latency from their business completely. With the SQL language emerging as the key enabler for the mainstream adoption of Hadoop, executing streaming SQL queries over Hadoop extends the platform out to the edge of the network, making it possible to query unstructured log file, sensor and network machine data sources on the fly and in real-time.

Organizations that deploy Big Data tools to analyze operational data soon learn staff’s expertise in SQL isn’t always helpful. New skills need to be learned.

Distinguished analyst Dan Kusnetzky of ZDNet discusses how SQLstream technology makes the SQL skills already found in most organizations useful for the analysis of streaming operational data.

Read full article HERE.

Unlike traditional relational database management systems (RDBMS), which persist data that can later be queried, SQLstream runs continuous queries on targeted data streams – enabling a real-time view of an organization’s situation.

Eric Kavanagh recaps SQLstream’s last session in The Briefing Room, held alongside analysts Mark Madsen and Robin Bloor- read full summary here.

SQL or NoSQL? In-memory or hard disks? Graph? These questions have been top of mind in recent years as developers and IT administrators check out new-age databases capable of handling scale-out data sets. SQLstream CEO and Founder Damian Black, together with 3 other executives from new data management technology companies, showed how they stand out in a hot market at GigaOM’s Structure:Data.

Read full article and watch full recording on gigaom.com HERE.

• SQLstream s-Streaming Big Data Engine Benchmarks at 1.35 Million Streaming Events Per Second per 4-core server  – Outperforming Twitter’s Storm Stream Computation Project with Significant Overall TCO Advantage

New York, NY | March 20, 2013– SQLstream Inc., the Streaming Big Data Company, announced today at GigaOM’s Structure:Data, the results of an independent performance benchmark which measured the SQLstream s-Server 3.0 Big Data Engine processing 1.35 million 1Kbyte records per second per 4-core commodity server, outperforming a comparable configuration based on the Twitter Storm distributed real-time computation system. SQLstream’s s-Server outperformed the Storm-based solution by a factor of 15x.

SQLstream’s s-Streaming Big Data Engine delivers action-oriented analytics, extracting operational intelligence in real-time from high velocity, unstructured log file, sensor and other machine-generated data. Streaming intelligence can be persisted, queried and replayed in Hadoop, with additional connectors to all major storage platforms and data warehouses.

The streaming Big Data benchmark was conducted by a large enterprise with a roadmap to stream unstructured operational data from multiple remote log and machine data flows at up to 10 million records per second for each installation. The benchmark requirement was to perform advanced time-series analytics over mobile network infrastructure records in order to predict potential service-impact problems. The benchmark projects that the s-Server platform would require just eight servers to scale up to 10 million records per second — versus an estimated more than 110 servers for the comparable Storm approach.

SQLstream s-Server 3.0 was able to demonstrate significant cost savings with dramatically lower TCO. The TCO savings came from a combination of reduced hardware and power consumption, the power and simplicity of SQL over low-level Java development, plus reduced maintenance requirements. Other factors influencing SQLstream s-Server’s TCO advantage came from its integrated Big Data platform architecture, ability to update on the fly as new data flows are incorporated, significantly faster implementation timescales using SQL for streaming analytics and integration, and automatic platform optimization for turbo-charged performance and parallel dataflow execution.

“SQLstream excels through the combination of its mature, industry-strength streaming Big Data platform, our support for standard SQL (SQL:2008) for streaming analysis and integration, plus a flexible adapter and agent architecture,” said SQLstream CEO Damian Black. “SQLstream s-Server is today’s clear streaming performance winner – with blazingly fast throughput, an ability to handle a wide variety of message types, sources and formats, and an efficient Streaming Data Protocol with compact optimized binary data formats.”

Advantages of SQLstream’s s-Server, the core element of the company’s s-Streaming Big Data Engine, as demonstrated in the performance benchmark project include:

• Scaling to a throughput of 1.35 million 1Kbyte records per second per four-core server each fed by twenty remote streaming agents.
• Expressiveness of the standards-based streaming SQL language with support for enhanced streaming User Defined Functions and User Defined Extensions (UDF/UDX).
• Deploying new processing analytics pipelines on the fly without having to stop and recompile or rebuild applications.
• Advanced pipeline operations including data enrichment, sliding time windows, external data storage platform read and write, and other advanced time-series analytics, all based on existing SQL standards.
• Advanced memory management, with query optimization and execution environments to utilize and recover memory efficiently.
• Higher throughput and performance per server for lower hardware requirements, lower costs and simple to maintain installations.
• Proven and mature enterprise-grade product with a validated roadmap and controlled release schedule.

All required modules used in the benchmark were integrated with s-Server 3.0, using 20 remote streaming agents connected per SQLstream s-Server instance each running on a four-core Intel® Xeon© server platform with RedHat Enterprise Linux.

About SQLstream

SQLstream (www.sqlstream.com) is the pioneer and innovator of a patented Streaming Big Data Engine that unlocks the real-time value of high-velocity unstructured machine data. SQLstream’s
s-Streaming products put “Big Data on Tap™ – enabling businesses to harness action-oriented and predictive analytics, with on the fly visualization and streaming operational intelligence from their log file, sensor, network and device data. SQLstream’s core V5 streaming technology is a massively scalable, distributed platform for analyzing unstructured Big Data streams using standards-based SQL, with support for streaming SQL query execution over Hadoop/HBase, Oracle, IBM, and other enterprise database, data warehouse and data management systems.  SQLstream’s headquarters are in San Francisco, CA.

• “Four For The Future: Upcoming Database Technologies” Session to be Held at GigaOM Structure:Data on March 21^st at 4:40PM ET

San Francisco, CA | March 14, 2013 – SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today announced that SQLstream CEO Damian Black will speak about innovative real-time Big Data technologies at GigaOM’s Structure:Data 2013, March 20-21.

Held on Pier Sixty, The Chelsea Piers in New York, and now in its third year, Structure:Data  gathers data scientists, business leaders and investors interested to learn more about the emerging business opportunities for Cloud and Big Data. Damian will speak on the invitation-only panel session Four For The Future: Upcoming Database Technologies at 4:40PM ET on Thursday, March 21.

Damian will be discussing the emergence of SQL for Big Data, and more specifically, SQL as a streaming language for low-latency, real-time operational intelligence in a Hadoop environment. Damian was selected for the panel because of his pioneering work in real-time and Big Data software. SQLstream’s s-Streaming products transform log, sensor and other machine-generated data into streaming operational intelligence in real-time. Its core streaming Big Data architecture is built on a massively parallel platform for processing high velocity machine data. Streaming intelligence can be persisted, queried and replayed in Hadoop, with additional connectors to all major storage platforms and enterprise data warehouses if required.

SQLstream CEO Damian Black has participated in many GigaOM events, including the first ever Big Data panel session at GigaOM Structure 2008. “It’s always a pleasure to speak at a GigaOM and to engage with an audience at the forefront of Big Data,” said Damian Black. “Real-time is the next wave of Big Data, and SQLstream is the future of true real-time business decisions. Real-time is only a streaming SQL query away.”

If you are a member of the press or analyst community and are interested in setting up a meeting with SQLstream at GigaOM Structure:Data, please contact Ronnie Beggs via email at ronnie.beggs@sqlstream.com.

About SQLstream

SQLstream (www.sqlstream.com) is the pioneer and innovator of a patented Streaming Big Data Engine that unlocks the real-time value of high-velocity unstructured machine data. SQLstream’s s-Streaming products put “Big Data on Tap™ – enabling businesses to harness action-oriented analytics, with on-the-fly visualization and streaming operational intelligence from their log file, sensor, network and device data. SQLstream’s core V5 data streaming technology is a massively scalable, distributed platform for analyzing unstructured Big Data streams using continuous SQL, with support for streaming SQL query execution on Hadoop and other enterprise database, data warehouse and data management platforms. SQLstream’s headquarters are in San Francisco, CA.

Resources

GigaOM Structure:Data 2013 agenda

GigaOM Structure:Data 2103 registration

SQLstream GigaOM Structure:Data 2013 Damian Black interview

In Part One of the Hadoop 2013 series, Merv Adrian points out how significant attention is being lavished on performance in 2013. In this second installment, the topic is projects, which are proliferating precipitously. One of the most frequent client inquiries is “which of these pieces make Hadoop?” As recently as a year ago, the question was pretty simple for most people: MapReduce, HDFS, maybe Sqoop and even Flume, Hive, Pig, HBase, Lucene/Solr, Oozie, Zookeeper. When the Gartner piece How to Choose the Right Apache Hadoop Distribution was published, that was pretty much it.

Since then, more projects have matured, and SQLstream is now part of of shortlist.

Read full article on gartner.com HERE. 

Hadoop 2013- Projects

Developed to Meet Demand from Universities and Supercomputing Centers

San Francisco, CA | February 20, 2013 – SQLstream, a pioneer of real-time Big Data, today announced an extension to its partner program that helps to fulfill the growing need for real-time data processing systems in supercomputing centers and academic cyberinfrastructure facilities.

The SQLstream University Partner Program (UPP) addresses educational institutions looking to expand research into streaming Big Data concepts by offering a non-commercial, royalty-free license to its market leading real-time analytics platform.

The Cornell University Center for Advanced Computing (CAC) headlines the growing list of UPP organizations using SQLstream’s streaming analytics platform to gain real-time insight into massive volumes of data. CAC is one of 17 partner institutions in the Extreme Science and Engineering Discovery Environment (XSEDE), a single virtual system that scientists can use to interactively share computing resources, data and expertise.

“Too many computing systems producing too many logs too quickly are unmanageable,” says Lucia Walle of Cornell University CAC. “SQLstream provided the real-time data mining system compatible with our local business logic that allows CAC to find events of importance using Sisyphus and queries, accurately simulate logs including the scale and pace of log activity, and monitor both simulated and actual system logs so that we can gain reliable insight into the cause and imminence of system failures before they actually occur.”

All UPP members have access to the full suite of SQLstream server technologies and have access to engineering support from SQLstream. Among many of the SQLstream University Partner Program benefits are:

• Access to a non-commercial, royalty-free license to the SQLstream technology;
• Access to a library of use cases and analytic algorithms;
• Discounted training and kick-off assistance to research projects;
• Discounted commercial license available for those Universities desiring to implement SQLstream in a production environment.

“One of our main goals as an organization is to accumulate business intelligence by adding research partners that can challenge and complement our existing solutions,” said Chris Clabaugh, VP of Business Development at SQLstream. “Streaming analytics and Big Data is seeing a lot of demand in the commercial and research space right now, so we are thrilled to be the streaming analytics technology supplier to university supercomputing centers.”

To learn more about UPP, please visit http://www.sqlstream.com/university-partner-program/.

SQLstream Inc. (www.sqlstream.com) makes systems responsive to real-time operational Big Data. SQLstream enables organizations to query their log, sensor and service data directly, and to share streaming operational intelligence with external systems, continuously and in real-time. SQLstream is built on a standards-based, distributed and massively parallel architecture, and uses industry standard SQL for the rapid analysis of high volume, real-time data streams. Standards mean lower costs, proven performance and seamless integration. Headquartered in San Francisco, CA, SQLstream is transforming the world of real-time, Big Data stream management.

Today’s edition of Information Management’s DM Radio Broadcast, The Future of Integration: ETL, CDC and IOA, had a great panel line up discussing the breadth of data integration issues in today’s world of Big Data, Cloud and traditional enterprise architectures. The session was hosted by Eric Kavanagh (Bloor) and Jim Ericsson (ex-Information Management, today was his last DM Radio after 5 years at the helm), and supported by Philip Russom (TDWI). SQLstream’s Damain Black was on the panel, with representatives from Denodo and Dell.

Extending the reach of Hadoop to the edge of your business.

A key observation emerged from the discussion, that the world of ETL and data integration is changing with the need for lower latency business operations. Most businesses are now global. There is not the window of opportunity for batched data management processes. Exploiting streaming data enables organizations to extend the reach of their Big Data storage platforms and data warehouses out to the edge of their business, connecting these platforms directly to the data sources. Streaming data can be filtered, aggregated and integrated with minimal latency, plus real-time operational intelligence extracted as the data stream past. This means businesses can be responsive to their real-time data without having to wait for data to be stored and processed.

Blurring the boundary between analytics and operations.

The emergence of low latency business operations is also blurring the boundary between traditional BI analytics and the world of business operations. The emergence of operational intelligence is the application of analytical techniques in real-time. The two are no longer distinct business functions, rather a seamless continuum from streaming data to real-time business value. The benefit is that businesses can now extract the information they need in real-time from arriving data streams, but still populate existing Big Data storage platforms and data warehouses, enabling existing business processes to function as normal.

The rising power of SQL queries over Hadoop.

Integrating data between systems implies a degree of structure, a common understanding of sink and source formats. The majority of integration platforms today assume SQL as the data management language. However, SQL as a language does not necessarily mean that the underlying data store has to be relational. And this is the game changer for Hadoop that will open up the existing enterprise market – the emergence of a true standards-based SQL query interface. Cloudera is already some way down this road with Impala.

So what’s the answer: SQLstream = Real-time data integration + operational intelligence + SQL.

I was listening to the BBC News on the Internet, and the technology section had two experts explaining Big Data to the layman. After some mutterings about it being ‘very large’ and ‘useful in genomics’, most listeners would have been left none the wiser. The BBC’s Science Editor, Susan Watts, had a bit of a better crack at it here. Our industry obviously has a way to go in explaining itself to the wider world.

Susan Watt’s BBC article of Big Data

Tackling the idea of ’very large’ first, and how Big is Big Data? Turns out, it isn’t, at least, not yet. EMA Research recently published a report; that debunked the myth of Big Data being Petabytes upwards. Authored by John Myers and Shawn Rogers, the report suggests that Big Data starts at 110GB, with typical use cases between 10 to 30TB.

But even at that size, why is the Big Data market still forecast to grow dramatically, with IDC predicting the market to be $24B by 2016? Surely this can’t all be put down to market hype and engineers’ excitement. Well, the answer lies in the ‘useful in genomics’. Not everyone has data of that scale or complexity, but most want to extract value from their data much more quickly and often in real-time. Big Data technology is aimed at making that easier, by removing many of the constraints of traditional data management and processing. For example, removing fixed schemas, processing all data, and generating answers on the fly, all with low cost, commodity hardware.

So Big Data Analytics is the killer app? Absolutely. Unfortunately, with today’s first generation Big Data platforms such as Hadoop, HDFS and even HBase, many are still struggling to analyze their data at scale and in real-time. That’s where technologies such as streaming and the (re-)emergence of SQL as a Big Data processing language is key. For example, Cloudera’s Impala is the first full frontal attack on mature Big Data market for larger enterprises. Real-time and SQL will help us bridge not only the technology gap, but also the ability to explain in layman’s terms how Big Data really benefit the person in the street.

Bloor Group’s Robin Bloor hosted SQLstream’s CEO Damian in The Briefing Room on Tuesday January 8th 2013. The webcast, entitled “Windows of Opportunity: Big Data on Tap” focussed on the emergence of both SQL and the streaming data platform as a key enabler for real-time Big Data solutions in an ever-maturing marketplace. You can watch the full webinar from the link below, but I’m going to focus on some of the topics arising from the online discussion between Robin, Damian and the audience.

It was an interesting discussion, covering Big Data and streaming in the wider context of enterprise deployments, but a number of important points were raised:

Hadoop is a data reservoir, not a real-time platform.

Many believe incorrectly that Hadoop is a platform for real-time low latency analytics.  It’s not. Hadoop is a multi-purpose engine but not a real-time, high performance engine. The parallelism of Hadoop is great for processing the data once it’s stored, but has high throughout latency.  However, with the integration of a streaming data platform for continuous data collection, analysis and streaming integration, Hadoop can be used as the active archive for a true real-time, streaming Big Data system.

Operational intelligence needs a Streaming Big Data Platform

The bulk of real-time operational intelligance today is derived from log and machine data, data generated by the Internet, Cloud infrastructure and applications for example. There are many log monitoring tools out there, and while very capable, we’re finding that SQLstream with our real-time streaming Big Data platform is being used to solve the high volume, high velocity, complex data problems that log monitoring tools are unable to address at an affordable price point.

The emergence of SQL for Big Data

The first phase of Hadoop and Big Data platforms saw the emergence of NoSQL data storage platforms, looking to overcome the rigidity of normalized RDBMS schemas. However, as the technology hits mainstream industry, the need for simpler, high performance and reliable queries is driving a resurgence in SQL as the de facto language for Big Data processing (see Cloudera Impala for example). What’s not apparent is that SQL is the ideal language for processing data streams using real-time, windows-based queries. The issue with normalization and rigid schemes is a non-issue for a streaming data platform – there are no tables, no data gets stored!

So in summary, streaming Big Data is the emerging technology for 2013. And SQL is the (re-)emerging technology as Big Data hits mainstream industry.  Processing real-time log and machine data streams is a key requirement today, but industry with sensor, M2M and telematics applications are catching up fast.

Dylan Janus from Information Today talks about the latest SQLstream release, 3.0. With improved speed and scalability, the flagship product introduces high performance distributed stream processing. Google BigQuery integration, and enhanced platform manageability and streaming application development.

“We have improved ways of processing machine data, so you can process much higher volumes and much more flexible and resilient processing of log file data,” Damian Black, CEO of SQLstream, tells 5 Minute Briefing.

Read the full article on DBTA.com here. 

The SQLstream Briefing Room webinar with Robin Bloor took place on Tuesday January 8th 2013.  ”Windows of Opportunity: Big Data on Tap”  highlighted how the evolving Big Data landscape needs technologies that enable a much bigger enterprise-wide picture, complete with multiple data streams that can be combined to show what’s happening in real-time. The speakers included:

• Eric Kavanagh, CEO, The Bloor Group, who hosted the event.
• Robin Bloor, Chief Analyst, The Bloor Group, who lead the online briefing
• Damian Black, President & CEO, SQLstream, discussing the emergence of streaming Big Data management as a key enabler for Big Data solutions, and how SQLstream is at the forefront of streaming innovation.

This was a very interesting and informative briefing on the emergence of streaming Big Data management and the use cases for real-time Big Data solutions.

Click here to watch the webcast …

Title: Windows of Opportunity: Big Data in Tap
Most business opportunities are moving targets these days, rendering static analytical solutions rather ineffective. Instead, organizations need technologies that enable a much bigger picture, complete with multiple data streams that can be combined to show what’s happening in real-time. And increasingly, companies need to analyze both traditional structured data as well as Big Data, including machine-generated data from all manner of enterprise systems.

Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor explain how a confluence of market forces has opened the door to a new analytical paradigm, one in which companies can leverage a vast array of data streams to pinpoint windows of opportunity as or even just before they appear. Bloor will be briefed by Damian Black of SQLstream, who will discuss his company’s analytical platform, which enables the management of dynamic information assets in much the way that traditional databases do for stored assets.

Justin Rowlatt, BBC World Service, explores how Big Data is not just a whole lot more thank the old “small” data, but a transformative technology that is set to change the way we do everything. From Formula 1 racing to targeted retail, the effective use of data is going to be a basis for competition going forward and a necessary addition to the toolkit of every company.

This evolution toward coexistence of the Hadoop framework, NoSQL and SQL approaches could mark a new step in big data’s maturation. As 2013 approaches, there is the possibility that big data may move from hot topic to practical reality.

SQL may have taken a punch or two in 2012, but it refused to go down for the count. Companies specializing in the alternative NoSQL and Hadoop side of things brushed up their SQL credentials this year. A prime example was Hadoop startup Cloudera Inc. It looked to enhance its SQL standing with Impala, a Hadoop software offering that supports interactive queries done in standard SQL.

Read the full article HERE

I have recently researched IBM® Infosphere Streams’ “Stream Processing Language” or SPL, after a pretty good talk at a recent SVForum SIG group meeting. Early users of this technology (which are really PS projects disguised as license sales) found that their Version 2.0 Stream Processing Application Declarative Engine (SPADE) programming language has been replaced by the IBM Streams Processing Language (SPL).

I suppose the research labs either changed interns for the summer or the products team have had changes in management. More seriously this is a natural outcome of productizing a customer software development project, paid for by the US Government, and then re-badged as a COTS product (commercial off-the-shelf product). IBM has time and again demonstrated their mastery of this jiu-jitsu move. It is truly impressive.

What we have done I believe is actually far harder, yet far simpler for the real users concerned with looking at and analyzing massive flows of streaming big data. From the start, we had a fundamental belief that our customers, partners, and end users, would need a familiar programming syntax and language. We would do all the difficult hard stuff “under the covers” as it were, so as to hide the complexities of aggregating and analyzing time-stamped streaming data over some time frame of relevance. We made a bet some 10 years ago that the deterministic and concise characteristics of SQL would continue to be a dominant force in the market; yet we undertook the very hard work of making difficult streaming data management appear easy to our end users and partners.

So, as IBM continues to revise / debug / enhance its various versions of SPL based upon various PS-led projects in the field, we view that as a great thing – for SQLstream and our customers. Our standard remains constant.

New SQLstream s-Server 3.0 enables faster response to operational Big Data by tapping into streaming log file, sensor and service data in real-time.

SAN FRANCISCO, CA, December 13, 2012 – SQLstream Inc. (www.sqlstream.com), a pioneer of real-time Big Data, today delivered the new generation of its streaming Big Data management platform. SQLstream s-Server 3.0 is the fastest and most scalable release of the company’s flagship product, introducing high performance distributed stream processing, Google BigQuery integration, and enhanced platform manageability and streaming application development.

SQLstream’s Big Data on Tap™ platform architecture is built from the ground up for real-time, streaming Big Data applications. The new release, with its real-time data collection, transformation and sharing capabilities, enables businesses to respond even faster to their operational Big Data. SQLstream s-Server 3.0 queries log file, sensor and service data in real-time, joins and transforms data streams using only the standard SQL language, and shares results continuously to outputs such as Big Data storage platforms. Streaming SQL enables high volume, high velocity applications for both structured and unstructured data to be built rapidly without having to resort to low level code development.

With s-Server 3.0, throughput performance is up to 10 times faster than that of the previous releases. The performance breakthrough is enabled by lock-free distributed processing for live data streams, a method pioneered by SQLstream and essential for real-time Big Data scalability. In addition, s-Server 3.0 brings faster integration, achieved through a new bi-directional connector for Google BigQuery, and faster development through new streaming SQL operators and added Windows support.

“Recent EMA research shows that Big Data environments consist of multiple platforms including structured (RDBMS) and multi-structured (Hadoop and other NoSQL) data stores – each one handling the processing that matches the strengths of the platform. This collection is called the Hybrid Data Ecosystem,” said John Myers, senior business intelligence and data warehousing analyst at Enterprise Management Associates. “Streaming Big Data solutions like SQLstream s-Server offer continuous integration with real-time analysis. This will be important in the area of operational intelligence analysis where low query latency is key.”

“SQLstream s-Server 3.0 addresses the operational Big Data problems that other log and network monitoring software solutions are unable to solve,” said Damian Black, SQLstream CEO. “Organizations want to be more responsive to their real-time Big Data, and that requires SQLstream’s low latency and raw performance, plus our ability to join and share data continuously between any data source and any destination storage platform.”

An early availability program for SQLstream s-Server 3.0 validated the new performance and integration capabilities across a range of industries including telecommunications, transportation, financial services and High Performance Computing (HPC). High performance log and machine data processing was a key requirement, met by SQLstream s-Server 3.0’s ability to detect real-time operational issues in high volume, high velocity data streams that were out of range for existing log monitoring systems.

“We had too many systems producing too many logs too quickly for any of our existing tools to process in real-time or otherwise,” says Lucia Walle, Cornell University Center for Advanced Computing. “SQLstream is the solution that scaled to monitor logs in real time for key patterns indicating imminent and undesirable conditions.”

To download a free trial and for more information about SQLstream s-Server 3.0, its technical capabilities, business benefits, and our Big Data on Tap™ approach, visit www.sqlstream.com/whats-new-in-3.

About SQLstream Inc.

SQLstream Inc. (www.sqlstream.com) makes systems responsive to real-time operational Big Data. SQLstream enables organizations to query their log, sensor and service data directly, and to share streaming operational intelligence with external systems, continuously and in real-time. SQLstream is built on a standards-based, distributed and massively parallel architecture, and uses industry standard SQL for the rapid analysis of high volume, real-time data streams. Standards mean lower costs, proven performance and seamless integration. SQLstream is headquartered in San Francisco, CA.

The move over recent years has been towards increasingly distributed processing of data, both in terms of the underlying model and in terms of the processing architectures available. More and more of this data is also streaming and SQLstream is focused on effective access to streaming big data …

Read the full article on Decision Management Solutions: http://jtonedm.com/2012/12/03/first-look-sqlstream/

I have a difficult time restraining my impulse to interrupt customers, vendors, employees, the janitor etc., when I hear them refer to ‘real time data’ when what is meant is the ability to ingest massive amounts of data and to interpret or analyze this in a short amount of time. Invariably they will say, “Well, this is clearly a ‘real time’ data problem” or the like. In short, they have confused acceptable low latency with streaming flows of data.

As the character Luke (portrayed by Steve McQueen, who many moons ago drove Vic Hickey’s Baja Boot in one of our Baja 1000 excursions, the Baja Boot being the precursor to the Humvee – but that’s another story) says in the classic 1967 movie Cool Hand Luke, “What we’ve got here is failure to communicate” – I submit  ‘real time’ can never be actually achieved or determined; it is very much akin to the intersection of Brownian motion with Heisenberg’s Uncertainty Principal – assuming you can catch / read / interrogate something in ‘real time’, both time has passed and the mere action of observing the object has caused the object to move away. That is, ‘real time’ can never be truly achieved or realized, an event is over by the time one tries to measure or quantify it. Low latency, or rather low latency in a time frame of relevance, is actually what these folks mean.

Streaming data, however, is constant, flowing, measurable, and accessible should one have the means to do so, SQLstream s-Server being a great tool for such purposes. Results from analysis of such streams are streams themselves, or can be quantized into discrete events which can then be used to trigger or visualize events of import, leading systems, administrators, executives, etc., to take appropriate actions. This needs to happen in a low latency time constraint; however, ‘low latency’ itself may have different meanings for different applications of these technologies.

Silicon Valley Comes to Oxford 2012 (SVCO 12) is an iconic entrepreneurship event now in its 12th year, that brings the most relevant technology entrepreneurs from Silicon Valley to Oxford, UK to engage with University of Oxford’s Saïd Business School students plus members of the wider Oxford entrepreneurial ecosystem. SQLstream CEO Damian Black is hosting and presenting sessions on both aspects – the trials and tribulations of getting a tech Big Data startup off the ground in Silicon Valley, plus insight into the current state and future potential of the Big Data market. There’s also the debate at the University of Oxford Student Union where Damian will be leading one side of the debate, but more on that in a separate blog.  Other speakers include senior representatives from Google, IBM and McKinsey. Click here for the full list of speakers.

What they don’t teach you in Business School on how to create and build a start-up company.

Damian’s workshop on tech startups focuses on a real-world perspective that is often overlooked in Business Schools. Most are aware of the importance of developing the right ideas, the right business plans, on building the right team and attracting the right capital. These are of course all useful steps but the reality is often very different.

Damian is leading a workshop that will cover the lessons and scars learnt during the early days of SQLstream and other start-ups, giving a real-world perspective as to what was easy, what was hard, what worked, what didn’t, and at the unanticipated pitfalls and the unexpected boosts that together made up the start-up journey of taking a company from crazy concept to everyday reality.

The full material and paper from the workshop will be published here on the blog over the coming weeks.

Also coming up in the next few weeks, The “Big Data” Janus Effect (and why is everyone might be looking the wrong way). Damian’s talk on Big Data at SVCO 12 will look at the intersection of Big Data and real-time, and whether the current emerging Big Data storage technologies are the beginning of a new era, or the final death throes of the static storage approach to business and operational intelligence that has persisted since the 70s.

Silicon Valley Comes to Oxford is on the 18th and 19th of November at the Oxford Centre for Entrepreneurship and Innovation.

Real-time Big Data means the streaming, continuous integration of high volume, high velocity data from all sources to all destinations,  coupled with powerful in-memory analytics. It’s a paradigm shift from conventional store and process systems that’s playing well here at the Intelligent Transportation Systems and Solutions World Congress here in Vienna.

(more…)

The latest installment in the datacentre mysteries – in which Hadoop has long been the prime suspect for writing off the relational database (RDB) – is out now. With fox-like cunning, [the NoSQL movement] devised a plan so that traditional databases could be slowly migrated onto this new model, without anyone noticing.

And they would have got way with it too, if it wasn’t for a young startup, SQLstream, sticking its nose in where it doesn’t belong. Damian Black, its CEO, might have accidentally disrupted the perfect disposal of RDB, by creating a real time analytical system that doesn’t need to number crunch big data. Instead, it intercepts live data streams, in real time, and creates insights from them a lot quicker. If they used it to analyse tweets about traffic jams, for example, they could avoid hundreds of millions of snarl ups on the motorway by tipping people off quicker than those ludicrous outdated ‘information’ notices they have on all our major trunk roads.

Click HERE to read the article in full.

GigaOM took their successful format for the Structure events to Europe this week for the first time and SQLstream’s CEO, Damian Black, was one of the key speakers. In a session with GigaOM’s leading Big Data writer, Derrick Harris, Damian discussed how real-time Big Data can only be achieved using continuous streaming integration, and how standard SQL is the perfect vehicle as a powerful language for the massively parallel distributed processing of live data streams.

As always, the GigaOM events were extremely well executed,and drew many of the main thought leaders and business leaders in the industry. For example, Ame Awadakkah CTO of Cloudera and Werner Vogels CTO of Amazon.

Real-time Distributed Big Data Processing

Damian’s talk was entitled ‘The Rebirth of Dataflow Computing: Real-time Distributed Big Data Processing’, and discussed how dataflow, an architecture for massively parallel distributed computing, is ideal for streaming Big Data integration and analysis. And SQLstream’s dataflow architecture harnesses the power of parallel dataflow using standard SQL as the data management language.

SQL is ideally suited to driving massively scalable applications based on a parallel, distributed processing architecture. In turn, this enables our customers to respond to each and every new record or piece of data as they arrive, in real time. So SQLstream is super low latency, unlike traditional data warehousing and Hadoop storage platforms, where data must be stored first and then queried. Even if you query faster, querying faster is not real-time.

We’re at the Intelligent Transportation Society’s World Congress in Vienna, Austria next week, Mon 22 – Fri 26 October. The theme for this year is ‘Smarter on the Way’ and this is the first year since 2009 that it’s been held in Europe. The full panoply of intelligent transportation issues is addressed, everything to do with software, hardware and firmware that improves the traveler experience, transportation network utilization and capacity optimization. However, the combination of real-time traveler information systems, wireless sensors, GPS and predictive analytics is a specific focus. And increasingly, telematics and V2I/V2V applications are represented.

We’re going to be exhibiting, but also presenting on a special interest panel session: Optimizing ITS from a Customer Service Perspective. Essentially looking at how new technology can be used to make a significant and noticeable difference to a traveler’s experience. Our focus is to bring to bear our experience of consumer and customer quality of experience monitoring from other industries, particularly telecommunications. Social media and Twitter also has a role to play, with significant interest is extracting semantic information from Tweets to detect serious traffic incidents and congestion as it happens. However, there are issues with relying on social media, as I’ve discussed here in a recent ITS International article, Social media mooted for traffic management.

Industries such as transportation, M2M and telematics are clearly leading the way in deploying real-time Big Data applications. And that’s understandable, it’s sensor networks and the Internet of Things that’s creating vast volumes of high velocity data where each event or record has value for only a very short period of time after its creation. If you’d like to understand more about streaming geospatial analytics from live vehicle GPS data, the video below presents a brief overview of a recent deployment.

Continuous ETL for Google BigQuery

There is another significant difference from the traditional business intelligence and real-time analytics market . The mainstream Big Data market is focused entirely on Business Intelligence, a one way function for  collecting data in a static store, and displaying the results on a graph or report. Visualization is also an important consideration for industry, but streaming integration is the first requirement, and the ability of the platform to support real mission critical applications a close second.

The Intelligent Transportation industry has all three requirements in abundance – vast volumes of fixed, wireless GPS sensor data that must be integrated seamlessly across the operational systems, mission critical operation as significant safety, environmental and traveller impact decisions are made as a result, and the abilty to visualize network flow, congestion and alarms.

So if you’re in Vienna, Austria next week, drop by our booth, P21.

Perhaps the highlight of Oracle OpenWorld last week, or at least, the most commented on by attendees at our booth, seemed to be Larry Ellison’s demo of Exadata and Exalytics – querying 10 days or so of stored twitter feeds with the hope of finding the best US athlete from the recent London 2012 Olympics to endorse a car company. This seemed to strike a chord with the audience. How many organizations employ a marketing analytics company to spend a vast amount of time poring over data to work out the top candidates for a marketing campaign? That said, would the CMO really go with a query result, or chose their favorite in any case?

Cloud and business applications were a focus, although as others have blogged elsewhere, despite 80+ acquisitions in the past few years, Oracle remains a database company. Major announcements / news included:

• Release of Oracle 12c (the ‘c’ for ‘cloud’), and the announcement of its first multi-tenanted and ‘pluggable’ databases got a few ripples of applause from the audience.
• Exadata X3 box, the in-memory machine with 22 raw TB of memory and a claimed 10X compression making a total of 220TB of ‘memory’ in a rack. Oracle claims this is 100 times faster that the Exadata Oracle launched in the last few years.

Streaming analytics and Twitter

Back to Larry’s Twitter example. Of course, this can be achieved easily as a streaming application in real-time. Semantic streaming is something SQLstream’s been doing for some time, taking unstructured data such as twitter, emails and texts and determining sentiment and aggregated scoring in real-time. Use cases include identifying traffic incidents on the road networks to augment geospatial analytsis of vehicle GPS data, and also in telecommunications, to better determine in real-time a customer’s true perception of their quality of experience for delivered services.

The numbers seemed impressive – Larry crunched nearly five billion tweets and 27 billion social media relationships. But breaking this down, is this really a Big Data problem? Five billion tweets, even over a one day period (I believe the demo was 10 days), is only 58,000 tweets per second. This is well in access of Twitter’s top peak loads during major events such as the Superbowl. But well within the capability of SQLstream’s real-time streaming Big Data platform, even on an entry level single server, 2-core machine. Of course, the complete solution architecture may include data storage platforms such as Hadoop or Oracle, where aggregated streaming results can be loaded and persisted in real-time, further crunched in the data warehouse, and historical analysis joined back with the real-time streams to help identify better any moving trends.

It was an interesting demo nonetheless, and one that really should be completed in real-time as a streaming problem. SQLstream’s ability to analyze and aggregate streams across in this case keywords and hashtags, provide geospatial and clustering analysis, as well as delivering raw and aggregated data as continuous streams to the backend storage platforms, makes this very achievable today.

On the show floor

Apart from the heavy footfall at the SQLstream booth, perhaps most notable was the increasingly uninventive marketing mechanisms used to persuade unsuspecting attendees to listen to product pitches based on the promise of winning a piece of Apple hardware. Surely marketing managers can think up something a bit more inventive than an iPad? The exception was the the speaking robot. Not sure if this was an exhibit floor attraction, although I saw it ‘chatting’ to passersby on the Wipro booth.

Contact us if you’d like to find out more about SQLstream and our streaming Big Data management platform.

We’re going to be exhibiting at Oracle OpenWorld with SQLstream next week. Looking forward to it following some great analyst briefings over the past few weeks and a lot of interest now from  companies looking for a real-time Big Data management platform where they can built there apps using standards-based streaming SQL.

In the beginning, or at least 18 months ago, there was Hadoop, and an increasing awareness that the Internet, sensors, M2M and cloud infrastructure was generating more data that existing platforms could manage.  The trend was Map-Reduce, NoSQL databases and a wide range of analytics companies claiming ‘real-time’. But querying faster is not real-time, even with the distributed power of Map-Reduce and more structured storage platforms such as HBase and Cassandra. Nor does Hadoop and related technologies address one key requirement – the real-time integration of high volume, high velocity and high variety data. Real-time, streaming integration is essential to making Big Data applications work.

Even more importantly, mainstream industry is looking to adopt real-time streaming technology, but does want to base their mission critical applications on high latency, non-standardised development frameworks.

This is where SQLstream comes in.  A real-time Big Data management platform for integrating live data, and performing real-time, in-memory analytics on the data as it streams past.  Industry and major companies have a real-time need, but prefer the use of standards such as SQL. The ability to lift existing SQL from their high performance analytics databases and data warehouses and drop the queries into a real-time streaming platform is a key.

So if you’re going, come and see us, we’re at #106 Moscone South.

This week we’re at the Intelligent Transportation Society of California’s Annual Conference & Expo in Sacramento. The conference is focussed on the adoption of advanced technologies to improve traveler mobility and heighten safety in California.  Attendees include specialists from government, industry and academia.

This week we’ll be demoing real-time traffic congestion analytics and visualization based on processing live vehicle GPS data. Play the video below to watch a short video of the live customer case study.

The real-time traffic analytics and connected vehicle programs are of particular interest to SQLstream.  SQLstream turns live data into real-time value, enabling industries such as Transportation, Telecommunications and Automotive Telematics to integrate and analyze all their real-time data on a single, streaming Big Data platform.

So if you are visiting Sacramento this week, we’d love to meet up.

SAN FRANCISCO, CA, August 23, 2012. SQLstream Inc, the leading standards-based real-time Big Data solutions provider, today announced the appointments of Glenn Hout as Vice President of Sales for the Americas and Chris Clabaugh as Vice President of Business Development. Hout and Clabaugh join the senior management team to build on SQLstream’s strong foundation in North America, and to expand SQLstream’s success into Asia Pacific and Europe.

Damian Black, SQLstream CEO, attributes SQLstream’s success to the rate with which industries such as transportation and telecommunications are adopting real-time, streaming Big Data technology. “As a result, we have seen a surge in customers seeking real-time Big Data solutions,” continues Damian. “Chris and Glenn bring passion, experience and leadership, as well as the existing relationships necessary to establish SQLstream as industry’s preferred partner for mission critical, real-time Big Data applications.”

Glenn Hout brings twenty-four years’ experience in the database and software applications industry with both early-stage and established software organizations. Glenn joins SQLstream from Big Data startup Algebraix Data, and prior to that, held management positions at Oracle, Information Builders and Hyperion solutions.

Chris Clabaugh joins SQLstream from Kabira Technologies (acquired by TIBCO) where he was Vice President of Business Development, overseeing ISV and OEM partnerships globally, managing branch operations in Japan, and opening new markets and geographical expansion in the Americas. Prior to Kabira, Chris held management positions at Collabnet, Progress Software, Allegrix and SCO.

Interested in Real-time Big Data solutions? Register here for a 60-day free trial of SQLstream s-Server. The download version is fully functional, and includes all adapters, drivers and development tools.

About SQLstream Inc.

SQLstream is the leading standards-based platform for integrating and analyzing streaming Big Data, forging real-time competitive advantage from live service, system and sensor data. SQLstream’s standards-based, distributed and scalable architecture uses industry standard SQL for the rapid analysis of high volume, real-time data streams. Standards mean lower costs, proven performance and seamless integration. With SQLstream, our customers are turbo-charging their Big Data environments for real-time, and responding with confidence to business exceptions based on accurate, up to the second information. SQLstream is headquartered in San Francisco and can be found on the web at www.sqlstream.com.

Media contact: Ronnie Beggs, +1 877 571 5775, pr@sqlstream.com

Datanami article on SQLstream for real-time Big Data solutions, reporting on a SQLstream application for real-time traffic analytics based on live GPS data.

“Providing real-time traffic updates is a practical and feasible application of the big data analysis technology that exists today. According to real-time streaming big data platform vendor, SQLstream, which aims to integrate and quickly analyze live data feeds,  “High-performance systems such as SQLstream Transport are able to transform large volumes of raw GPS data into real-time actionable information.” The transformation in question took place on the roads of Venezuela in the form of SQLstream’s ETL Connector for Google Big Query….”

Read the full article on datanami here http://bit.ly/M9ps1b

SQLstream, the leading platform for real-time Big Data integration and analytics, today announced it has joined the Google Cloud Platform Partner Program as a Technology Partner with its release of the Continuous ETL connector for real-time Big Data integration with Google BigQuery.

Continuous ETL for Google BigQuery

SQLstream’s continuous ETL is the key building block for powerful real-time Big Data solutions that integrate vast volumes of real-time streaming Big Data with historical trend information. SQLstream provides streaming integration for structured and unstructured data in real-time and on a massive scale, overcoming the poor scalability and high latency issues that are typical with traditional batch-based solutions. With SQLstream, organizations can act instantly on new information as it arrives, improving operational efficiency and driving new revenue opportunities.

“The Google Cloud Platform Partner Program enables us to integrate our product offerings for real-time Big Data with the power of the Google cloud platform,” said Damian Black, SQLstream CEO. “SQLstream is the only real-time, streaming Big Data management platform that is built on the ANSI and ISO SQL standards, making it the perfect real-time complement for Google Big Query. Our customers benefit from the power of the Google Cloud Platform while utilizing the same SQL skills to perform streaming Big Data integration with real-time analytics.”

The continuous ETL integration has already been deployed by Grupo Intech in Venezuela as an integral component of a SQLstream and Google BigQuery solution for detecting road network traffic congestion in real-time using GPS data. “Understanding the quality and coverage offered by each of our GPS data providers is essential,” said Marcelo Ricigliano Cantos, CEO of Grupo Intech. “Google BigQuery is updated continuously and in real-time by SQLstream, and generates on-demand confidence indicators as to GPS data quality and reliability.”

“To help customers get the most out of our cloud platform products,” explains Eric Morse, Head of Sales and Business Development, for Google’s Cloud Platform, “we work closely with technology companies, like SQLstream, that provide powerful complementary solutions integrated with our platform.”

Google’s Cloud Platform products enable customers to implement:

• Cloud app solutions, such as mobile apps, social apps, business process apps, and websites, using Google App Engine and Google Cloud SQL.
• Cloud storage solutions, such as high-end backup and recovery, active archiving, global file sharing/collaboration, and primary SAN/NAS, using Google Cloud Storage.
• Large-scale computing solutions, such as batch processing, data processing and high performance computing using Google Compute Engine.
• Big data solutions, such as interactive tools, trend detection and BI dashboards, using Google BigQuery and Google Prediction API.

About SQLstream Inc.
SQLstream is the leading standards-based platform for integrating and analyzing streaming Big Data, forging real-time competitive advantage from live service, system and sensor data. SQLstream’s standards-based, distributed and scalable architecture uses industry standard SQL for the rapid analysis of high volume, real-time data streams. Standards mean lower costs, proven performance and seamless integration. With SQLstream, our customers are turbo-charging their Big Data environments for real-time, and responding with confidence to business exceptions based on accurate, up to the second information. SQLstream is headquartered in San Francisco and can be found on the web at www.sqlstream.com.

Contact: Ronnie Beggs, VP of Marketing, +1 877-571-5775

SQLstream has joined the Google Cloud Platform Partner Program as a Technology Partner with the release of our Continuous ETL connector for real-time Big Data integration with Google BigQuery. Continuous ETL solves the real-time performance issue for Big Data storage platforms. Although very capable of processing large datasets once the data has been stored, Big Data storage platforms are not designed to process real-time streaming data.

Continuous ETL for Google BigQuery

SQLstream’s Continuous ETL connectors enable SQLstream to integrate and analyze vast volumes of live, real-time Big Data, and to update the Big Data storage platforms immediately as input data arrives. With existing ETL solutions, the data warehouse and Big Data storage platforms are updated in batch mode and are therefore always out of date. Continuous ETL updates continuously and in real-time.

Real-time, integrated Big Data solutions

The continuous ETL integration was carried out by Grupo Intech as an integral component of a SQLstream and Google BigQuery solution for detecting road network traffic congestion in real-time using GPS data.

GPS records are collected from several specialist data providers. Each GPS record contains the position, direction and speed of a vehicle. SQLstream collects GPS records from all the providers in real-time and turns the live GPS data feeds into real-time traffic flow information, and by applying a variety of congestion detection algorithms, is able to generate a map of the current traffic congestion positions, plus additional information as to the extent or severity of the problem.

SQLstream calculates real-time traffic analytics and displays the results in real-time on a map-based display. Roads are color-coded based on the average traffic speed relative to the posted speed limits for each road segment. Colored push-pins appear on the highways to provide further information on the potential problem, such as the average speed for every minute over the previous 15 minutes.

Continuous ETL and Google BigQuery

So how does Big Query help? GPS data is of variable quality, and also, as the data is available from many different providers, the network coverage for each provider can differ significantly.

Google Big Query is used to store the real-time traffic information produced by SQLstream, and to generate a confidence index on the coverage and quality, and therefore the reliability, of the incoming GPS records. Effectively, Google Big Query is helping Grupo Intech answer the question – do we have sufficient valid data for the real-time congestion information to be reliable.

SQLstream’s Connector for Google Big Query is used to deliver real-time analyzed data using a Continuous ETL operation. With continuous ETL, streaming data is continuously acquired, cleaned, conditioned, transformed and then periodically appended to the BigQuery table, for example, every minute or every 500,000 events. Therefore the data in Big Query is always up to date and accurate.

Here we see the Big Query table, which holds a minimum of 3 months of enhanced and interpolated data for ad-hoc data mining. The period can be extended if needed for improved accuracy.

Continuous ETL – more than real-time

SQLstream’s analysis of the road network is based on individual 10m road segments. Typically GPS records are not available for every vehicle for each and every road segment, so SQLstream interpolates missing road segment data in real-time, and continuously appends those results as well into the Google Big Query table – storing a complete real-time map of traffic flow for the entire road network at a level of granularity of every 10 meters. Therefore unlike traditional ETL tools that tend to offer only a basic mapping function, SQLstream delivers significantly more information into Google Big Query than is available in the source data.

How does it work

SQLstream allows the user to zoom in and center the geographic map to an area of interest. This action defines a bounding box and the coordinates of the bounding box are used in this first Big Table query to extract the number of 10 meter road segments within the bounding box. The action of defining the bounding box in SQlstream  launches a query similar to the following example:

select count(*) from road_elements
where
  "reLatitude" between 7.7109920000000001 
                   and 9.8525100000000005 AND
  "reLongitude" between -65.754776000000007 
                    and -62.958756000000001.

This query retunes the number of road segments within the bounding box (in this example, there are 191,866 road segments) which is then plugged into the query executed by BigQuery in order to calculate the percentage of road elements for each GPS data provider actually used for calculating the road network coverage within the bounding box.

select vendorID, float(integer(count(*) / 191866 * 1000)/10) as Confidence
from (select vendorID, reID from sample.july2nd
where 
 float(reLatitude) between 7.7109920000000001 
                       and 9.8525100000000005 AND
 float(reLongitude) between -65.754776000000007 
                        and -62.958756000000001
group by vendorID, reID)
group by vendorID order by Confidence desc

The query results are returned to SQLstream which displays the answer to the user. Big Query is ideal for these types of geospatial coverage metrics. There is a high volume of meaningful data stored in the Big Query table and the calculation of coverage is available immediately. By choosing a variety of bounding boxes, it is possible to obtain coverage metrics a country-wide basis, on a metropolitan area basis, and on for a much smaller problem area.

Conclusions

In conclusion, we have demonstrated how continuous ETL is the basic building block for integrating real-time and historical data. Continuous ETL enables offline data storage to be kept up to date directly from real-time applications. And unlike conventional ETL solutions that simply map data between systems, SQLstream’s Continuous ETL connector for Google Big Query delivers aggregated data, analytics as well as augmented and interpolated data values.

We’ve been exhibiting at Structure 2012 in San Francisco, where our CEO Damian Black was speaking on dataflow architectures for massively scalable real-time Big Data computing. In fact, this was a milestone for us as Damian was on the very first Big Data panel at the first Structure event in 2008.

Dataflow is a technique for parallel computing that emerged from research in the 1970s. It’s based on graph-based execution models where data flows along the arcs on a graph and is processed at the nodes. It was decades ahead of its time in an era when hardware was expensive and real-world requirements for massively parallel, low latency computing architectures were not required in the mainstream. However, dataflow as an architecture has found its place and time, with the emergence of Big Data volume, real-time low latency requirements, commodity hardware and low cost storage. Dataflow is driving the architectures for today’s real-time big data solutions.

Click to view Structure 2012 presentation video

SQLstream adopted the principles of dataflow as the basis of our architecture for SQLstream s-Server. Our adapters turn any data source into a live stream of data tuples which are combined, aggregated and analyzed by the SQLstream s-Server platform. SQLstream has added one essential feature to data flow – the use of SQL as a dataflow management language. SQL has been used for some time as the language of choice for relational database management systems, and in this context is getting a bad press in light of new structures for Big Data storage and NoSQL queries. However, SQL is powerful, declarative (therefore applications can be built easily, quickly and cheaply) and is a natural, powerful paradigm for processing streaming dataflows. The benefit is extremely low latency with the ability to process massive volumes of live data over an unlimited number of servers – exactly the requirements of real-time Big Data. In fact, this is the only architecture capable of processing real-time Big Data streams. With real-time requirements now in the 20 to 100 million events per second range, power, scalability and low latency are key.

Diagram 1: Dataflow architecture for real-time streaming Big Data computing

The SQLstream s-Server architecture concept is illustrated in Diagram 1. As a dataflow architecture, each node is a streaming SQL statement – a continuous SQL query, processing arriving data over a moving time window (time windows can be from 1 millisecond for ultra low latency requirements, to months or even years where comparison against long term moving averages is required, for example, Bollinger bands). Why is this important? Well, it’s the only approach for low latency, real-time solutions, as information flows out of the system as soon as input data arrives, that is, the high latency of batch-based approaches such as Hadoop Map-Reduce is removed completely.

Mozilla Glow: Real-time download monitor with SQLstream and HBase

Damian presented a simple example of SQLstream and parallel dataflow in action. Mozilla’s Glow application is a continuously updating download counter for the Firefox 4 browser when it was released. The application used SQLstream s-Server to collect live download statistic from all the download servers worldwide. Download records were processed and aggregated in real-time and displayed on the Glow visualization map, illustrating exactly how many copies of the browser had been retrieved. SQLstream s-Server also provided a continuous ETL operation into Apache Hbase, storing aggregated and filtered records for further in depth analysis. Click here to watch the application in action.

Finally, and in contrast to Structure, we also attended a Gartner session last week with Merv Adrian and Svetlana Sicular, which sought to bring some sense of perspective to Big Data. This was really a reality check as to the current maturity levels of the Hadoop Big Data platforms and the effort required to deploy. The wider adoption across industry in general will require significantly more mature products and applications, particularly around the OPEX costs for deployment, security concerns and ability to deliver business intelligence all consumers in an large organization. The recommendation was to use an integrator such as Cloudera or Hortonworks. Mainstream organizations are looking at the Hadoop / Big Data approach, but many do not currently see either a use case or a reason for adoption. It was interesting to hear a perspective that didn’t need to be buzzword compliant, and presented a positive yet realistic perspective on wider adoption.

We’re at Sensors Expo this week, showcasing in the Big Data & Analytics Pavilion. This is the first year the event has included a specific area for real-time Big Data solutions for sensor networks.

Real-time control in a Big Data World

SQLstream CEO, Damian Black, presented on Real-time Control in a Big Data World. The presentation focused on the increase of sensor data and the emergence of the “Sensor Internet”, plus the applications required to collect and analyze streaming sensor data, and to drive real-time actions and updates. In particular, addressing the emerging real-time Big Data challenges in this area driven by wireless and GPS technologies, M2M applications and V2V/V2I.

Click here to view Real-time Big Data integration and analytics for sensor networks

Real-time streaming data integration for Big Data

It’s clear the primary challenge is not managing the data volume per se, or even delivering real-time operational intelligence, rather it’s the more fundamental issue of real-time streaming data integration. How can such huge volumes of data from many different sources and locations be integrated into the operational platforms, and how can the issues of multiple operational siloes be overcome to provide an integrated real-time control platform.

Interestingly, these are the exact same issues SQLstream addresses for the Big Data and Hadoop world in general – getting data in, getting data out, connecting existing data stores in real-time, and delivering real-time in-memory analytics on the data as it streams past:

• Real-time streaming data integration of any data source and between existing storage platforms and operational systems
• Real-time streaming monitoring and analytics on the arriving and streaming data
• Scalability through parallel distributed processing of processing pipelines

The importance of geospatial analytics

Geospatial analytics is a key requirement in the sensor data market. Big Data analytics in general is about one dimensional problems, usually the correlation of similar events, or the correlation of events over time. The geospatial dimension is the key difference between Big Data platforms for the “Sensor Internet” and the wider IT / machine data applications. Fortunately this has been a feature of SQLstream for some time, and central to many of our customer deployments. For example, real-time traffic analytics from GPS data, and real-time seismic monitoring.

Glue Conference 2012 , Denver CO, at the end of May was a great conference, well attended, knowledgeable participants and is the only conference I know that looks at gluing cloud and mobile applications together with a developer focus.

There was the usual wave of NoSQL, cloud storage, cloud platforms and Hadoop presentations, as you’d expect, but also with some interesting keynotes as well. Ray O’Brien, CTO for IT at NASA. talked about the evolution of Nebula and OpenStack at NASA, and James Governor from Redmonk, talking about the evolution of historical analytics.

From our perspective, the strength of the show was in making physical rather than logical connections. Both partnerships and potential customer interest in building real-time Big Data applications, and how SQL has been repurposed as an API for streaming Big Data, moving it forward significantly from its roots as static data management language.

Real-time, streaming Big Data

Relational Streaming for real-time Big Data scaling

Our CEO, Damian Black, presented on real-time streaming Big Data, both as a real-time alternative to Hadoop, and also as a complement to add real-time responses and streaming integration to existing Hadoop installations. One question we were asked several times was why SQL? A good question. This isn’t a religious debate about the language by any means, and if we had opted to build a Big Data batch storage and analytics platform (e.g. like Hadoop), we would have gone a different route.

However, when it comes to processing streaming tuples in real-time, a standard SQL approach has two big advantages over all others. First, with the extension of the SQL WINDOW operator to process streaming data over fixed time windows, both structured and unstructured data can be processed painlessly without having (no pun intended) to define a static schema and without the need for any coding whatsoever. In effect, SQLstream processes streams of arriving tuples over time windows and pushes out the results to other systems.  Similar in concept at least to Hadoop, although Hadoop is purely batch-based, processing static files and pipelining sets of tuples through low level Map-Reduce functions.

Click on the title slide to view the real-time Big Data presentation

However, the second benefit is equally important. Streaming SQL queries include standard operators such GROUP BY and PARTITION. These provide the best clues possible to a query planner capable of automating the dynamic scaling of streaming pipelines over vast numbers of servers. This gives a reliable and controllable mechanism for Big Data scalability without the need for hardcoding server allocation hints.

Real-time at GlueCON

The strength of the real-time track at GlueCON was encouraging. It was interesting though that the term ‘real-time’ is now about as over used as ‘Big Data’, and about as poorly understood. For SQLstream, it’s the streaming integration of any and all data sources with in-memory analytics, processing streams at millions of events per second. For some other vendors, it appears real-time drops off at significantly lower rates and numbers of connections!

Next stop, GigaOM Structure in San Francisco

Next stop GigaOM Structure in San Francisco, June 20 / 21 at the Moscone Center. Visit us there if you’re attending.

We’re at ITS America Annual Meeting, National Harbor, Washington DC  this week (see the team in action below), the yearly opportunity for the US intelligent transportation community to get together and discuss how IT and technology can be used to better serve travelers, industry and government. Real-time, the Smart City and Vehicle-2-Vehicle communication have been the trend topics.  Bill Ford previously said “the car will be a rolling set of sensors” has echoed here.  Martin Thall, VP Telematics at Verizon talked about the issues of getting the data up from the vehicle, and how 4G / LTE will be key due to the large volume of fast data, however the pricing and subscription model for 4G services is currently one of the hurdles to wider telematics adoption.  Essentially real-time Big Data requirements for mission critical applications, the area where SQLstream excels.

Chris Vein, deputy white house CTO discussed the need for open standards, and for government to find ways for private industry to participate more.  The idea of “government as platform, encouraging the “lean startup” and to offer “smart disclosure” of data to encourage information sharing and real innovation.

Why is my GPS Travel Time always wrong?

Ursula Burns, Xerox CEO captured exactly the need for SQLstream s-Transport. How come, every morning when I drive to work, does my GPS always tell me the journey will take 37 minutes, and it has yet to take 37 minutes?  Accurate and reliable real-time Travel Time is the first problem SQLstream solved with the SQLstream s-Transport product.  Using GPS data to monitor traffic flow accurately and in real-time, and present correct end to end journey time information throughout the journey, not just at the beginning.  (More on SQLstream s-Transport here).

I’ll update and publish a final report on the show  these next final few days. If you’re attending ITS America Annual Conference, National Harbor, Washington DC, please drop by Booth #631.

This week I’m attending an interesting conference at UC Berkeley called the “Berkeley conference on Streaming Data”.  The organizers are primarily astronomers and statisticians, but the talks discuss issues and solutions to streaming data problems across a wide selection of scientific areas and engineering applications.  Real-time streaming Big Data applications presented included oceanography biology genetics, reading handwriting, astrophysics, particle physics, recommendation engines for social media, and inevitably, real-time fraud detection from live data feeds.

I presented on a deployment of SQLstream as a Dynamically Scalable Cloud Platform for the Real-Time Detection of Seismic Events. Based on work with UCSD seismologists, SQLstream has been deployed to detect significant events in data collected from a large grid of seismic sensors. A large-scale data infrastructure (the OOI/CI) provides raw signal data over an AMQP message bus.

Plot of Seismic Events

SQLstream monitors live seismic data feeds in real-time, applying heuristic algorithms that look for patterns indicating earthquakes. The live system scales dynamically across multiple servers in a cloud environment based on the current demand. You can view the presentation here.  I also blogged previously on the application here.

In conclusion, I have two main observations from the conference so far (it continues until Friday). The first is that the majority of fields in science and technology appear to have a Big Data and often a real-time Big Data problem.  Secondly, the extent of the innovation and computer science resources dedicated to solving these problems.  In particular for this conference, developing algorithms for data analysis and machine learning (that is automatic pattern recognition) that work on streams of flowing data.  It’s clear that traditional data management and even Big Data batch-based methods don’t work when you need continuous results from dynamic data. And the amount of data is huge.

The Text Analytics Summit in London this week was an opportunity to catch up on the latest trends and state of the Text Analytics market.  An interesting couple of days with a few themes emerging.

Firstly, Big Data.  Not entirely unexpected, but almost every presentation referred to Big Data in some shape or form.  In part this was referring to the volume of data to be processed, but primarily in the context of databases for the storage and processing of unstructured data of any volume.

Although not discussed explicitly, there’s obviously a search for business models that work.  Most applications were B2B platforms, sold as a package of product, services and consultancy, enabling organizations to better mine text data for market and competitor intelligence.  However, some were seeking to monetize through subscriptions to information feeds.

For SQLstream, we presented on the use of real-time text analytics for improving incident detection and prediction.  In particular, the use of real-time Twitter and text messages for identifying Quality of Experience issues with IP content services, but also the use of Twitter for improving real-time incident detection in transportation networks.  And in line with the rest of the conference, we did our bit for Big Data, describing how real-time streaming integration and analytics can be built on unstructured data analytics as an integrated real-time Big Data and Hadoop platform.

Joining real-time structured and unstructured data feeds for better accuracy and reliability from your operational intelligence, and the Text Analytics Summit, 2012, London.

Three IT trends have emerged over the past year – Big Data, real-time and the importance of unstructured data. Taking the latter first, there is an increasing awareness that much of the data we have available to us today is unstructured (Cloudera amongst the many claiming 80% of all data is unstructured).  Unstructured data includes text messages, documents, tweets emails and video content. There’s also a growing industry for tools and software that perform unstructured data analytics – primarily text analytics using semantic modeling, tagging and subsequent analysis.

The past year has also seen Big Data and Hadoop emerge from the rarefied atmosphere of California’s Silicon Valley into mainstream IT.  Driven by statistics such as 90% of all data available today has been generated in the past two years, Big Data as a functional area for primarily unstructured data is here to stay, and is effectively supercomputing lite for the masses.

The need for real-time streaming data management

However, the real-time trend is less well served today by either Hadoop or by the currently available tools and software for unstructured data analytics. Real-time is about the need for immediate detection and response – turning data sources into live data feeds, and processing the data on the fly, then loading batch based distributed platforms such as Hadoop as an output data stream.

‘Stream Reasoning’

I’ve also seen the term ‘stream reasoning’ used to describe the real-time processing of unstructured data, although this is still an area that is less well developed and understood than the more mainstream text analytics from stored data.  ‘Streaming Reasoning’ is the ability to process and respond to semantic knowledge about tweets, messages and other social media interaction in real-time, on the fly. The diagram below illustrates how a semantic modeling library has been plugged into a real-time streaming pipeline in SQLstream – the example is based on SQLstream’s GATE UDX but any library with reasonable performance and a query response API can be plugged in.

Unstructured data feeds, such as text messages and tweets, are streamed through the semantic tagging UDX and library, with the output of this stage being real-time streams of semantic tagged data.  The data can then be analyzed and frequency charted in real-time.

Text Analytics Summit, 2012, London

I’ll be speaking on this topic at the  Text Analytics Summit, 2012, London.  I’ll be discussing how to combine streaming reasoning (admittedly, mostly Twitter messages) with structured data, with the objective of improving the overall accuracy and reliability of the resulting operational intelligence.  I’ll be using a couple of examples – customer experience management for IP content services such as VoIP and VoD, and also improving the accuracy and reliability of traffic congestion information and travel time information – how can text analysis of tweets and messages help to pinpoint the severity of road network traffic problems.

Look forward to seeing you there, or if you can’t make, I’ll be blogging on the highlights next week.

Last week SQLstream sponsored and CEO Damian Black presented at Structure Data in New York, a conference exploring “the technical and business opportunities spurred by the growth of big data”.

It’s clear that Big Data has moved on considerably in a very short space of time. From the Silicon Valley, 101 world of Java developers and Hadoop, into the mainstream wider business world (but still with Hadoop!).

Some themes emerging from the conference:

• The basic need to deliver high performance, massively scalable computing infrastructure as data volumes grow exponentially. It’s clear that the pain from structured and unstructured data is driving different approaches at different stages in the data management lifecycle – better visualizations, better cleansing and filtering, and a better understanding of the appropriate analytics tools that are most applicable at each stage.
• The emergence of the SQL layer. It’s clear Hadoop has its strengths and is here to stay. It’s effectively ‘supercomputing lite’ and given today’s data volumes, is just the tool for the job. However, there are a couple of trends emerging. First, is it actually necessary to store all the data, when much of it is obviously not of interest? Second, once the initial analysis of both all structured and unstructured data is achieved, there’s an emerging layer above Hadoop that’s looking very structured.  Both these functions are looking much more SQL-like.
• Real-time, low latency analytics. Hadoop is not, nor does not claim to be, a low latency, real-time data management platform. There is a well-defined business need to analyze log file, sensor and network data in real-time (sub-second to a few minutes latency), but also to stream the arriving data through to Hadoop for further analysis. Obviously this layer needs to as scalable, if not more so, than the underlying Hadoop platform.

Damian’s presentation Structure Data focused on relational streaming – massive-scale parallel data processing using SQL, generating real-time results from streaming input data. The talk described relational streaming as a standalone real-time management layer, and also SQLstream integrated with Hadoop as the streaming layer in the Big Data stack (you can also read the GigaOM report in the presentation here).

GigaOM reports on Damian Black, SQLstream CEO, talking about streaming Big Data at Structure Data, New York,March 21 – 22. In the talk entitled “Streaming Big Data: Millions of events per second”, Damian discussed the similarities between Hadoop Map/Reduce and the parallel, distributed architecture used for streaming data processing in SQLstream.

Click here to read the GigaOM report.

Visit our new website to find out more about real-time Big Data applications

Big Data is here to stay. The breadth of the term Big Data may change as it becomes as much a marketing imperative as the ‘Cloud’ word, but the requirement for ‘supercomputing lite’ processing for the non-supercomputing world of enterprise data is a must have.

The rise of Big Data has happened in parallel with the emergence of real-time operational intelligence, and the extension of real-time analytics into the world of real-time updates and process control. Much of the recent interest has focussed on how these two worlds merge into a single complementary solution.

The NoSQL BigData platforms offer massively scalable, resilient data processing over commodity hardware. Ideally suited to scaling large scale data problems over hundreds or thousands of servers. However, platforms such as Hadoop do not support, nor were designed to support, real-time streaming data processing and analytics. Their forte is the batch-based, highly scalable, store-compute loop of map/reduce.

That’s where SQLstream comes in. SQLstream collects and conditions real-time updates from sources such as log files, sensor networks and GPS events, and both integrates streaming data into and from Big Data stores, but also generates real-time analytics from the data as they stream past. The SQLstream architecture also has parallels to that of map/reduce. SQLstream uses Relational Streaming, which is a paradigm for processing streaming Big Data tuples using standard SQL queries. SQL offers strong potential for automatic optimization and distributed parallel processing of streaming data. Whereas platforms such as Hadoop execute batch queries over stored tuples, SQLstream and Relational Streaming executes continuous queries over arriving data.

We’re also at Structure Data this week in New York, where our CEO, Damian Black, will be presenting on the wider area of streaming Big Data and massive scalability. However, if you are attending, visit us for a demo of the ‘millions of events per second” program, and a demonstration of massively parallel stream processing on an Elastic Compute Cloud.

SQL is a declarative language – a SQL query is a specification for the result, it’s neither a recipe nor a program to produce the results. A traditional relational database query returns a set of rows, the ResultSet. A streaming SQL query in SQLstream returns a stream of rows. That is, the ResultSet may never end. In a traditional relational database query, all the rows are fetched, and the ResultSet scans them. With a relational streaming query, the result rows do not exist as yet – as time goes by they come into existence as arriving data are processed.

However, just like any relational database, the SQLstream stream computing Server has two main components:

1. The query engine or planner, calculates the most efficient plan to produce the requested results – this is the query plan.
2. The data engine or kernel, executes the query plan to produce the results. The scheduler controls the execution process.

Streaming dataflow graphs

Executing a query means computing the results from the inputs. For a streaming query that means processing the rows in the input streams as they arrive. The execution is organized as a dataflow graph, that is, a mathematical directed graph of nodes and arrows, where the nodes represent elementary operations on data, and the edges into and out of a node represent the input and output data streams. In effect, an assembly line that produces results, where the nodes are the machines or stages on the line.

A query plan defines one of these data flow graphs. The executor runs the data through the graph: it is responsible for executing the nodes, where each node consumes rows arriving on input edges and produces rows on the output edges. Of course each output edge is often the input of a downstream node.

Multiple, connected query plans

In a traditional static database, each query plan is independent and transitory, and operates against persistent tables and indexes. In a relational streaming platform, the query plans last forever and are interconnected. Although streams are used just like tables in SQL, they are not persistent, in fact they have no contents at all, and can be as rendezvous points that accept input rows and pass them on to their output consumers.

Now if the data flow graph were a physical system — say, a collection of transparent plastic straws with colored water flowing through them — then all the processing would be happening simultaneously. However, for the software abstraction of the streaming straw pipelines, it’s not practical or necessary to run all the nodes at the same time. It is the scheduler that manages this network of interconnected query plans, and when and how to execute each node.

In a traditional, static database, the result of a query is a set of rows that are computable all at once. The executor can give good performance by running one node at a time, pushing batches of rows through the graph. A streaming database is different. When the inputs are streams of recent events, arriving in real time, it’s important to produce the outputs fast enough so that the result rows are timely.

The execution works by pushing outputs, not by pulling inputs, and it means executing several nodes at the same time, whenever possible. This requires a finer management of the execution objects and the ability to schedule parallel execution on the nodes.

Parallel scheduling of stream execution

In SQLstream, multiple, interconnected query plans are being executed at the same time. Together they constitute a large dataflow graph in which each node is a mini data processing machine that performs a simple operation on its input data, and passes its output to the next node.

The scheduler is responsible for managing the interconnected dataflow graph. It keeps track of the status of each node: at any moment, some are running, some are ready to run, some are waiting for more input, some are waiting for their output to be consumed. Each node is allowed to run for a quota of time, and where possible nodes are selected to execute in parallel as separate threads.

The SQLstream scheduler may not want to be fair: some branches in the graph (some streams) may be more important, some may need high throughput, some may need lower latency. The application designer decides, and the SQLstream scheduler delivers.

Next time …

This is the first is a series of blogs discussed both the principles and practical examples of parallel stream execution. The next blog in the series will look at some real world examples, and how parallel execution is essential to deliver both high throughput and low latency requirements.

One of the great advantages of SQLstream as an analytical platform is that it uses the most popular, standardized language for data analysis, SQL. SQLstream worked to make only the minimum number of extensions to SQL necessary to encompass the streaming data paradigm, so that most streaming SQL pipelines look almost indistinguishable from SQL for reading static relational data. This enables data analysts to leverage virtually all of their existing SQL skills in the streaming context.

Similarly, SQLstream felt it was important to make the streaming environment feel familiar and productive to application developers as well, so SQLstream supports the standard JDBC interface for using streams, again with just the minimum extensions necessary to encompass streams.

This post assumes a basic familiarity with JDBC and its main components: connections, statements, and result sets. First we’ll look at these in their usual tabular context, then see what it takes to extend the model to streaming data. All the data items come from the SALES example schema that comes with SQLstream.

Reading JDBC Data

In a database world, the pattern for reading data is quite standardized: Connect to the database, execute a query, and read each row that comes back until there are no more rows. For example,

Connection c = getConnection();
try {
     Statement s = c.createStatement();
     ResultSet rs = s.executeQuery(“SELECT * FROM SALES.EMPS”);
     while (rs.next()) {
           System.out.println(rs.getString(“NAME”) + “ “ + rs.getString(“EMPID”));
     }
     rs.close();
     s.close();
     c.close();
} catch (SQLException se) {
}

This simple example loops through the entire EMPS table, printing the name and employee ID number for each row in the table, then finishes. The key for this model is that the result set is finite. Even if this were a very large table, the loop would eventually process every row. So you can handle reading the data from a table as a monolithic step in a sequence of procedures.

The Challenge of Streams

In a streaming data environment, however, you have to change a couple of your basic assumptions. In particular, I have found three Rules of Streaming that dictate how to write client code:

1. There always might be more data.
2. You never know when the next row might arrive.
3. The rate of the rows matters.

The same pattern shown earlier for reading finite tables will work for streams,  as long as you don’t expect your application to do anything else. For a streaming ResultSet, the next() method only returns false when the stream closes. In many applications, that might never happen, or at least might not happen for weeks or months or longer, so clearly you cannot simply wait for each stream to end.

This is particularly critical in an application such as SQLstream Studio, where a developer needs to be able to edit the definitions of objects and at the same time be able to watch data flowing in existing streams. These streaming data views – known as Inspect windows – have to function in an event-driven, multitasking environment. There can be a virtually unlimited number of them active at any given time, along with editors, console views, and other dynamic content. So nothing should block the updates of other views. And for a little added complexity, Studio also needs to be able to handle non-streaming items as well, so preferably the same code should handle either tables or streams.

Studio also has to deal with a number of other requirements that are fairly unique to the development environment, such as how to manage updating a human-readable window of maybe 20 or 30 rows at a time from a stream that might be flowing at many thousands of rows per second. For now, we will just focus on the tasks common to handling streams in any application environment.

Reading in the Background

As with any long-running task, the solution involves partitioning the work into threads. Because of Rule #1 (“There always might be more data.”), you should just assume that reading from a stream needs to happen in a background thread. There is essentially never a scenario in which you want to wait for an entire stream to be read before proceeding to the next task.

To some extent how you implement your background stream-handling tasks will depend on the environment you are working in. Variations in the data rate and the required responsiveness of the application might cause you to make some different choices.

One good rule of thumb is to do as little processing as possible in the stream-reading loop. Slow processing of incoming rows can result in “back-pressure” in the data pipeline. As a result, it’s best to read rows from the ResultSet as expeditiously as possible, handing off the data to other threads for processing.

Here, for example, is a simple thread to read rows from the SALES.BIDS stream:

class BidReader extends Thread
{
      @Override
      void run()
      {
            try {
                 Connection c = getConnection();
                 Statement s = c.createStatement();
                 ResultSet rs = s.executeQuery(“SELECT STREAM * FROM SALES.BIDS”);
                 while (!interrupted() && rs.next()) {
                       // read columns and put into work queue for processing thread
                 }
                // close rs, s, and c
            } catch (SQLException se) {
            }
      }
}

Note that the loop doesn’t depend solely on the ResultSet’s next() method, but also tests whether the thread has been interrupted. You probably wouldn’t use this exact mechanism (I prefer to override Thread.interrupt() and set a boolean flag), but it shows you succinctly that you need to be aware of more things than just whether there is more data to read.

Another thing to keep in mind is that ResultSet.next() blocks until either more data arrives or the stream closes. That’s yet another reason to have this happening in a background thread.

You’ll notice that code snippet references a “processing thread.” That’s because of Rule #2 (“You never know when the next row might arrive.”). It’s generally good to decouple the reading code from the processing code. One easy model is to have the reader read each row, then add the row to a shared queue structure, where the processing thread can pick up the rows and process them asynchronously. This minimizes the chances of a slow processing step causing the reader to slow down and potentially push back up the stream. To the extent possible, you’d like your reader to be sitting and waiting for the next row when it arrives, rather than constantly trying to handle a backlog of incoming rows. If you have to have a backlog, you want it to be in your application, not in the stream server, because that will slow everyone down, including other applications trying to read from the same streams.

Such decoupling doesn’t really make sense in a pure database application, since there will be a finite number of rows to process, so your goal is to minimize total processing time. If there are 1,000 rows, it doesn’t matter whether you read them all in one second, then spend an hour processing, or read them over the period of an hour, processing as you go. The total amount of data read and processed is the same.

In the streaming world, it is important to read the data as quickly as possible in order to maximize the overall throughput of the system. That’s Rule #3 (“The rate of the rows matters.”) coming into play. The data throughput of a given stream is gated by its slowest reader. Part of the contract your client needs to fulfill is to not bog down the system for other clients.

QoS and service level monitoring has always presented a challenge for telecommunications companies. With the increase in uptake of IP voice and video services, the vast data volumes generated, and the lack of an end to end view, make monitoring the service experience in real-time increasingly difficult.

In this blog I’m looking at the core building blocks of a real-time IP service monitoring solution by using a much simplified view of a real-time application. Diagram 1 illustrates the basic problem – how to monitor an IP service when the end to end view is only possible by piecing together large volumes of events from many different sources – the core network provider’s network, the home network and the cable modem, and the service providers platforms.

In SQLstream we capture each event stream in real-time. Applications are built as streaming pipelines – unlike a traditional database solution, where event data must first be stored and then processed, SQLstream streams the data through processing views, capturing, combining, filtering, aggregating and applying analytics to the events streams, without having to store the data.  This enables real-time operational intelligence with extremely high volume performance with very low latency.

The first views in the pipeline capture the data streams. A declaration for an external data feed is shown below, the real-time MyEvent stream, where source of events is the external system agent or integration adapter.

CREATE OR REPLACE STREAM MyEvent
( "eventName" VARCHAR(10),
  "eventSeq" BIGINT,
  "eventVal1" INTEGER,
  "eventVal2" SMALLINT,
  "eventVal3" BIGINT )
DESCRIPTION 'source of events';

The raw MyEvent data is first filtered, searching for the events of interest. As illustrated in the code example below, these initial views tend to be as simple as possible in order to maximize reuse – the simplest being a SELECT STREAM * FROM WHERE statement. Streams can be combined, grouped or joined in a single view, or a single view provided per stream, or both.

CREATE OR REPLACE VIEW RawEvents 
    AS SELECT STREAM * FROM MyEvent 
WHERE "eventName" = 'RawEvent';

Diagram 2 illustrates the concept of the streaming data pipeline, using a simplified example for exception detection. The SQL view illustrated above is the Stream Capture #1 view in the diagram. The use case is built on a real world example, raising an exception if a number of events of a particular type or value are detected within a specified time window.

The second view in the pipeline, Stream Processor #1, is shown below. In this example the view is responsible for the basic processing of the stream, counting the number of events that occur within a time window, in this case 180 seconds.

CREATE OR REPLACE VIEW CountedEvents AS 
SELECT STREAM *, 
   COUNT("eventName") OVER win AS "eventCount",
   FIRST_VALUE(RE.ROWTIME) 
          OVER win AS "firstEventTime",
   FIRST_VALUE("eventSeq") 
          OVER win AS "firstEventSeq" 
FROM RawEvents 
   AS RE WINDOW win 
   AS (RANGE INTERVAL '180' SECOND(3) PRECEDING);

The final stage in this particular processing pipeline is the detection of the alert.

CREATE OR REPLACE VIEW FlagTriggerEvents 
    AS SELECT STREAM *, 
    "eventCount" >= 3 AS "alert" 
FROM CountedEvents;

It would of course be possible to include all processing in a single view. However, maximizing reuse of views is a major consideration when building a stream processing application. The example is to illustrate how a pipeline can be constructed, where each view can have any number of consumers. For example, any number of Rule views can read from the Stream Processor #1 view, and any number of views can read directly from the stream capture view.

The application includes significantly more sophisticated integrations, features and analytics than illustrated here. For example:

• Multiple rules
• Recording and forwarding the events responsible for the generation of the alerts
• Detect escalation
• Detect clearance events
• Join with alert history to identify exceptional events that deviate significantly from historical norms

These use cases are important components of a complete solution, and I’ll be providing examples in subsequent blogs, explaining how these have been implemented.

For Big Data, 2012 has started where 2011 left off, with a plethora of reports, articles and blogs. Interestingly, most still begin with the question “what is Big Data”. It appears ‘Big Data’ as a market is broadening its footprint far beyond its open source and Hadoop origins. My favourite new term in this quest for delineation is “Small Big Data”. (Isn’t that just “Data”?)

The most interesting trend for us is streaming Big Data processing and analytics. Edd Dumbill, O’Reilly Radar, talks about this as one of the “Five big data predictions for 2012”, “Hadoop’s batch-oriented processing is sufficient for many use cases, especially where the frequency of data reporting doesn’t need to be up-to-the-minute. However, batch processing isn’t always adequate, particularly when serving online needs such as mobile and web clients, or markets with real-time changing conditions such as finance and advertising.”

The real-time use case is an obvious one. If you need to respond or be warned in real-time or near real-time, for example, security breaches or a service impacting event on a VoIP or video call, the high initial latency of batch oriented data stores such as Hadoop is not sufficient.

However, there is also an emerging discussion on the storage of Big Data for big data’s sake. This is the blind collection and storage of data without due consideration as to how it’s going to be used. Dan Woods talks about this in his recent Forbes article “Curing the Big Data Storage Fetish”. The data will never create value without analysis, and little thought has been given to increasing analytics capacity.

There are many vendors emerging for the historical analysis of Big Data repositories, either on the Hadoop platform, or on platforms from the other large scale data warehouse vendors. However, there are very few vendors in streaming Big Data analytics space, and even fewer products with the maturity, flexibility and scalability to process Big Data streams in real-time.

Streaming Big Data analytics needs to address two areas.  First, the obvious use case, monitoring across all input data streams for business exceptions in real-time. This is a given.  But perhaps more importantly, much of the data held in Big Data repositiories is of little or no business value, and will never end up in a management report. Sensor networks, IP telecommunications networks, even data center log file processing – all examples where a vast amount of ‘business as usual’ data is generated. It’s therefore important to understand what’s being stored, and only persist what’s important (which admittedly, in some cases, may be everything).  For many applications, streaming data can be filtered and aggregated prior to storing, significantly reducing the Big Data burden, and significantly enhancing the business value of the stored data.  At least until we understand why we’re trying to store everything.

Event report from ITS California,  Annual General Meeting in Long Beach, Nov 13 – 15, 2011.

ITS California’s AGM is SQLstream’s local intelligent transportation event, and an opportunity for public and private sector companies to discuss those issues specific to the state of California. The event has grown in size significantly over the past few years, with new organizations attending, all keen to contribute and discuss different perspectives on the various transportation problems. ITS-CA was established in 1994 as a not-for-profit organization with the remit to foster the adoption of ITS technology across the state. Its funding is a mix of public and private industry.

California has some acute transportation issues. For example, in Los Angeles County, the number of new residents is set to increase traffic on already congested roadways by an estimated 39 percent. Roadway expansion in this period is set to increase by only 3 percent, resulting in congestion levels rising by more than 200 percent in the next 25 years. Obviously addressing these issues ha been the focus of the Californian transportation industry as a whole, but ITS-CA serves as a focus point once per year to discuss in a wider forum.

The theme of this year’s event therefore (not unsurprisingly) was “Discovering Keys to the Next Decade”, with session focused on:

• Support for connected vehicle deployments.
• Transportation systems efficiency, for example reliable travel time and pricing.
• Analyzing the success of traffic light systems and how close we are to the ‘continuous green’ vision.
• Improving traveller information systems (of specific interest to SQLstream).
• The effectiveness of schemes such as car sharing.

Although there’s always an over-arching theme or direction that emerges, and in this case it was the increasing presence of the private sector, and how private sector innovation is helping to drive the adoption of new technology and approaches. This was a common topic at the ITC-CA in Berkley last year, and has continued to grow. It was also a trend that was raised frequently at the recent ITS World Congress in Orlando. Long may it continue.

The Tutorial blog series helps SQLstream developers build streaming SQL applications. This blog is the third and final part of the Geospatial Visualization tutorial. The first blog in the series set out the streaming use case for connecting SQLstream to a Google Earth visualization, and described the initial steps required to capture the data and create a display list using Rails. The second part of this tutorial presented the core of the application – how to render the display list. And in this the concluding part of the visualization tutorial, the final key element is discussed – how to get the data flowing.

Getting the Data Flowing

The last step is to tie SQLstream and Postgres together. First we need to give SQLstream the credentials to access Postgres. Create a new file:

$QUAKES/quake.postgres.properties

1 URI=jdbc:postgresql://localhost/quakekml_development 
2 DRIVER=org.postgresql.Driver 
3 CONNPARAMPREFIX=dbConn_ 
4 dbConn_databaseName=quakekml_development 
5 dbConn_user=USER 6 dbConn_password=PASSWORD
7 dbConn_applicationName=SQLstream TableReader Adapter

On lines 5 and 6 insert the Postgres user and password you set up (same as in $QUAKES/quakekml/config/database.yml). Create a directory under $SQLSTREAM_HOME/plugin called jndi (if it’s not there already), copy your quake.postgres.properties file there and restart the SQLstream server.

linux> cd $SQLSTREAM_HOME/plugin
linux> mkdir jndi
linux> cp $QUAKES/quake.postgres.properties jndi/
linux> cd $SQLSTREAM_HOME
linux> bin/sqlstreamd

 1 SET SCHEMA '"WebFeed"';
 2 
 3 CREATE OR REPLACE FOREIGN DATA WRAPPER "TableUpdate"
 4     LIBRARY 'class com.sqlstream.plugin.tableupdate.TableUpdateStreamControlPlugin'
 5 ++++LANGUAGE java
 6 ++++DESCRIPTION 'adapter for doing insert/update/merge/delete to an external database';
 7 
 8 --
 9 -- Create SQL/MED foreign server and foreign stream
10 --
11 CREATE OR REPLACE SERVER "Postgres_TableUpdate"
12 FOREIGN DATA WRAPPER "TableUpdate"
13 OPTIONS (
14 ++++connParams 'quake.postgres',
15 ++++sqlDialect 'Postgres 8.x',
16 ++++pollingMillis '5000',
17 ++++commitCount '1000',
18 ++++commitMillis '2000')
19 DESCRIPTION 'Postgres database with visualization';
20 
21 --
22 -- Display list for quake events
23 --
24 CREATE OR REPLACE FOREIGN STREAM "QuakeEventsDB" (
25 ++++"SQLS_opcode" CHAR(2) NOT NULL,
26 ++++"SQLS_chg" VARBINARY(32),
27 ++++"id" INTEGER options("insert" 'skip', "update" 'skip'),
28 ++++"when" TIMESTAMP,
29 ++++"lat" DOUBLE,
30 ++++"lon" DOUBLE,
31 ++++"mag" DOUBLE,
32 ++++"created_at" TIMESTAMP,
33 ++++"updated_at" TIMESTAMP
34     )
35 ++++SERVER "Postgres_TableUpdate"
36 ++++OPTIONS (
37 ++++++++TYPE 'tableUpdates',
38 ++++++++MASTER 'true',
39 ++++++++updatesTable 'quake_events')
40 ++++DESCRIPTION 'table updated with quake events';
41 
42 --
43 -- Pump quake events into display list
44 --
45 CREATE OR REPLACE PUMP "1000-QuakeEventsPump" STOPPED
46 DESCRIPTION 'pump from "SmallQuakesDay" view to "QuakeEventsDB" foreign stream' AS
47 ++++INSERT INTO "QuakeEventsDB" (
48 ++++++++"SQLS_opcode", "when", "lat", "lon", "mag",
49 ++++++++"created_at", "updated_at")
50 ++++SELECT STREAM 'IN',
51 ++++++++q.ROWTIME,
52 ++++++++CAST(SUBSTRING("point", 1, POSITION(' ' IN "point") - 1) AS DOUBLE),
53 ++++++++CAST(SUBSTRING("point", POSITION(' ' IN "point") + 1) AS DOUBLE),
54 ++++++++"mag",
55 ++++++++CURRENT_TIMESTAMP, CURRENT_ROW_TIMESTAMP
56 ++++FROM "SmallQuakesDay" as q;

linux> sqllineClient < viz.sql

$QUAKE/start-pump.sql

1 ALTER PUMP "1000-QuakeEventsPump" START;

linux> sqllineClient < start-pump.sql