Blog

Real-Time Analytics with Stream Processing for Operational Intelligence and the Internet of Things

This week saw the publication of the results for a comparative real-time performance benchmark between Apache Storm and SQLstream Blaze. Using the WordCount example shipped with Hadoop and Apache Storm, we were interested to see just how quickly each could process records.

As it turns out, pretty quickly for Blaze, around 4.6 million records per second per 8-core server, less so for Apache Storm. The 4.6 million number is more or less equivalent to processing the Complete Works of Shakespeare (a little under 1 million words according to the Internet) fives times every second. Although why you might want to do that, I’m not sure. The difference in performance between SQLstream Blaze and Apache Storm was even more impressive as the WordCount benchmark does not expose one Storm’s major weaknesses – no native concept of time-based processing over time windows. Which happens to be a particular strength of SQLstream, and all real-world use cases for streaming analytics require this capability.

That said, the key point here is not about the fastest performance, it’s about what that means in real terms. The cost of Big Data and real-time systems has come into focus more over the past year. Although Big Data technologies utilize commodity hardware, when you need 100+ boxes to get any reasonable performance, that can become expensive. Faster performance per server means less servers.

The Cost of Performance for stream processing

Throughput performance offers confidence of future scalability but also translates directly into the monthly and lifetime costs for the solution. The average bare metal cloud server is around $500 – $2000 per month depending on the specification and I/O bandwidth.

ROI Metric Apache Storm SQLstream Blaze Blaze ROI
Hardware Cost / Month
Processing 5 million words per second using bare metal 8-core Cloud servers at $500/month
121 servers, $60,500 / month 2 servers, $1000 / month SQLstram Blaze 60X reduction in infrastructure cost versus Apache Storm.

Even though storage is less of a consideration with a stream processing platform, why deploy 100+ servers when the same work could be carried out on two or three? This means a significant reduction in server costs (60X reduction even considering just the simple WordCount benchmark), but also much simpler manageability and platform stability going forward.

Time to Value.

Time to value, system stability and agility for new requirements are also front of mind for CIOs when deploying Big Data platforms. Time to value is the time taken to get to an operational system from scratch. It’s swings and roundabouts with Hadoop and stream processing frameworks such as Storm. On one hand, anything is possible, an important consideration when processing unstructured machine data, but on the other, it takes time and money to build from scratch, often at the expense of solution reuse and the ability to change for new requirements. The WordCount example is trivial in both SQLstream Blaze and Storm, but in real-world scenarios, such as covered in a previous independent benchmark between SQLstream and Storm, time to value is an important consideration.

ROI Metric Apache Storm SQLstream Blaze Blaze ROI
Development Effort
From download to operations, based on a customer 4G network performance monitoring app.
6 months
(180 days)
1 week
(5 days)
SQLstream Blaze delivers robust operational systems 30X faster.

This is where the power of SQL for stream processing comes to the fore – powerful analytics, quickly, and a stable platform – but also pre-built adapters, integrated real-time dashboards for streaming analytics, out of box integration for continuous ETL and stream persistence with Hadoop HDFS and HBase, plus a range of other RDBMS and data warehouses.

In summary, performance does matter. Stream processing offers scalability for systems at the junction of fast data and big data. But stream processing platforms are not all equal. As Shakespeare said “Time travels at different speeds for different people” (As You Like It). For SQLstream’s customers, it gallops, and we can keep up.

Real-time to Action for Stream Processing

SQLstream Blaze outperformed Apache Storm by 113X using the industry-standard WordCount benchmark. For enterprises using Blaze, the result means streaming analytics from Big Data in motion using only a fraction of the servers required by open source stream processing frameworks, and therefore a fraction of the cost. Why… Read more →

Security Internet of Things

Cybersecurity and the Internet of Things are increasingly uncomfortable bedfellows. We’ve blogged before on the the security gaps that already exist as a result of connecting yesterday’s technology to the Internet. A recent article by Colin Wood published on govtech.com goes several steps further and brings us up to date. The… Read more →

Extreme data stream processing performance

The stream processing paradigm differs from the traditional storage-based data management paradigm with which we grew up. Stream processors are fast as they are in-memory (although this is not unusual these days), process data streams record-by-record as they arrive over time or record-based windows using continuous queries (which never… Read more →

Mobile-Map

Big Data is also about faster results, streaming analytics and real-time actions (as low as millisecond latency) in the case of stream processing, with faster batch operations in the case of Hadoop (a few hours). Fast data also means a different set of data quality issues – which must… Read more →

Real-time to Action for Stream Processing

There’s certainly a vast range of different IoT API, connection protocol technologies and data formats. At first glance, this makes device to device communication tricky, particularly as the Internet of Things encompasses all vendors and technologies. However, the Internet of Things is not necessarily a direct vendor to vendor… Read more →

Real-time to Action for Stream Processing

Bloor Group has launched its Big Data Ecosystem Directory to help their audience and customers research the most promising and innovative software vendors in the Big Data space. The Directory includes  live demos of the products, covering every aspect of the Big Data Ecosystem, as well as the outputs of… Read more →

September 15, 2014 by in In the News
Real-time to Action for Stream Processing

A large variety of commercial and open source event processing software is available to architects and developers who are building event processing applications. These are sometimes called event processing platforms, complex-event processing (CEP) systems, event stream processing (ESP) systems, or distributed stream computing platforms (DSCPs).
Distinguished analyst Roy Schulte from… Read more →

August 27, 2014 by in Stream Processing

Streaming analytics is the latest thing in Big Data: streaming analytics, simply put, enables organizations to leverage their fast Big Data in real-time, improving operational efficiency and customer satisfaction, while enabling new real-time revenue streams. Alex Woodie takes a look at The Forrester Wave™: Big Data Streaming Analytics Platforms, Q3… Read more →

Customer experience management (CEM) has piqued the interest of the communications industry, particularly as service providers seek innovative solutions to drive competitive advantage. CEM solutions offered by companies like Amdocs, Alcatel Lucent, Ericsson, Nokia and Oracle have evolved in recent years from those that independently focus on individual customer… Read more →

Real-time Big Data Cost of Performance

Data processing technologies perform at different rates, making the Total Cost of Performance a hot topic for feasibility studies concerning Big Data tools including traditional databases, Hadoop and stream processors. The reason is simple. Any storage-based technology must store the data first before they can be queried and processed,… Read more →

July 3, 2014 by in Hadoop
Real-time to Action for Stream Processing

Log analytics has been around for a while but until recently, “real-time log analytics” usually meant
a) slow answers (in many minutes or even a few hours)  and
b) low volumes (data arriving at a few thousand records per second).
Things, needless to say, are changing (could be the recent uprise in… Read more →

Customer experience has long been a tricky area for Telcos. Not that the data hasn’t been there, but it’s been distributed across different organizational and data silos, and making use of it has been challenging. In fact, the lack of a complete customer view has driven partial insights based… Read more →

300x200-Blog-BigData

A distributed data management architecture is an essential requirement for real-time Big Data applications such as managing IoT sensor and machine data payloads. Smart services for IoT applications will require low latency answers, multiple servers and distributed processing for scalability, plus built-in redundancy for resilient, 24×7 operations.
It’s also important… Read more →

SQLstream Blaze support for Apache Kafka merges standards-compliant SQL stream processing with Hadoop Big Data platforms on a single real-time, distributed messaging infrastructure.
SAN FRANCISCO, CA – May 29, 2014 – SQLstream Inc., the leading standards-based stream processing software company, today announced support for Kafka, Apache’s messaging framework. Kafka… Read more →

On Thursday, April 24, SQLstream hosted a webinar exploring the potential of the Internet-of-Things. With a focus on monetization, the event expanded on harvesting real-time value from IoT services, discussing technology requirements, security concerns and likely directions for commercialization.
So what is the Internet of Things? To many, it’s about connected devices,… Read more →

 “The 3Cs of the Internet-of-Things: Conditions, Concerns, Commercialization” will run on April 24, 2014
SAN FRANCISCO, CA – April 17, 2014 – SQLstream, Inc., the Big Data Stream Processors company, today announced that it will host a webinar exploring the potential of the Internet-of-Things. With a focus on monetization, the event… Read more →

SQLstream’s Stream Processor to Deliver the Next Generation of Real-time Gaming Analytics  
SAN FRANCISCO, CA – April 1, 2014 – SQLstream Inc., the Big Data Stream Processors company, today announced that The Workshop has selected SQLstream for real-time gaming analytics. The Workshop, a leading US gaming studio and creator of… Read more →

The Internet of things has been on everyone’s lips (and pen) lately, as the biggest new source of money, customer product innovation, investment opportunities, and Sci-Fi-worthy stories. The truth is the value of the Internet of things only comes from the astounding mass of data it’s bound to produce, and… Read more →

What goes around surely comes back around, which in the world of data is often called lifecycle management. Very few organizations have ever formalized and implemented such a grandiose practice, for only until recently has the concept become seriously doable without great expense.
Eric Kavanagh of Inside Analysis writes about a new,… Read more →

APIs are creating a new age of intelligent machine-to-machine (iM2M) analytics apps aided by SaaS. M2M data is everywhere, but individuals don’t want the flood of data — just the pint they need when they thirst for it. Distinguished market researched Kishore Jethanandani talks about how grossly underutilized data, often… Read more →

With the O’Reilly Media’s Strata conference opening one week ago, a lot of big data companies announced new products and looked for the spotlight. In this review published by CNN Money, distinguished analyst Derrick Harris of GigaOM goes through the most notable events in Big Data and lists SQLstream… Read more →

SQLstream 4.0 brings unparalleled real-time performance, Apache Storm integration, enhanced real-time visualization and new industry-specific StreamApps toolkits to the leading standards-based Big Data stream processor.
SAN FRANCISCO, CA – 11 February 2014 – SQLstream, Inc., the Big Data Stream Processors company, today announced the availability of SQLstream 4.0, the new… Read more →

February 13, 2014 by in Press Releases

January 14 Webinar with EMA Industry Expert, SQLstream CEO, and InfoArmor CTO, Will Share Expert Insights on how Enterprises Can Extract Value from Machine Data with Streaming Analytics
SAN FRANCISCO, CA – January 8, 2014–SQLstream Inc., the streaming Big Data platform for real-time operational intelligence, today announced that CEO Damien… Read more →

San Francisco, CA– SQLstream, Inc., the leading standards-based streaming Big Data analytics platform, announced today that Database Trends and Applications (DBTA) has listed SQLstream as a trend-setting product in real-time Big Data management and analytics. The Trend-setting Products in Data for 2014 honors organizations with a commitment to technology innovation and… Read more →

December 16, 2013 by in In the News

SQLstream StreamApps are fast-start templates for real-time streaming machine Big Data applications.  Each StreamApp is a library of components for a specific operational business process. In a Big Data industry typified by a lack of standards and high development costs, StreamApps takes SQLstream’s standards-based SQL platform for streaming operational… Read more →

 

SQLstream was named as one of the top trend-setting technologies for data management for 2014 by the Database Trends and Applications (DBTA) magazine.
Two of the most pressing concerns that organizations face today are the need to provide analytic access to newer data types such as machine generated data, documents and graphics,… Read more →

December 6, 2013 by in In the News

Just back from the Silicon Valley Comes to Oxford (SVCO), an invite-only event at the Said Business School, University of Oxford University in the UK. The aim of the event is to provide insight to the Business School graduates on how to start, scale and run high-growth companies. The speakers… Read more →

SAN FRANCISCO, CA – 14 November 2013 – SQLstream Inc., the streaming Big Data platform for real-time operational intelligence, today announced StreamApps, fast-start templates for real-time streaming machine Big Data applications.
 
Each StreamApp is a library of components for a specific operational business process. In a Big Data industry typified… Read more →

Veracity and SQLstream team up to deliver actionable Operational Intelligence for real-time call rating, performance management and fraud detection from streaming CDR and IPDR data
SAN FRANCISCO, CA – 12 November 2013 – SQLstream, Inc., the Streaming Big Data Company, today announced that Veracity Networks, an award-winning internet and telecommunications… Read more →

Click on image to listen to podcast
On October 4, 2013, CEO Damian Black briefed members of the Boulder Business Intelligence Brain Trust (BBBT) on the topic “From Big Data to Real-time Value: Streaming Operational Intelligence.”
Esteemed analyst Claudia Imhoff interviews Damian in a short podcast on Big Data, SQL and the… Read more →

On Friday, October 14, 2013, Damian Black, CEO, and Ronnie Beggs, Vice President of Marketing, of SQLstream, Inc. briefed members of the Boulder Business Intelligence Brain Trust (BBBT) on the topic “From Big Data to Real-time Value: Streaming Operational Intelligence.”
Distinguished BI Expert Paul te Vraak comments on the SQLstream… Read more →

October 10, 2013 by in In the News
monitor+dials

SQLstream s-Visualizer provides active graphical dashboards over streaming time-series data, mapping real time to human time without refresh
San Francisco, CA | October 8, 2013– SQLstream Inc., the Streaming Big Data Company, announced today the world’s first visualization dashboard product for streaming high-velocity data. SQLstream s-Visualizer provides active, graphical insights… Read more →

Oracle OpenWorld Logo

SQLstream to Demonstrate Real-time Big Data Solutions for Traffic Analytics and the Internet of Things in the Big Data and Business Analytics Industry Showcase
San Francisco, CA | September 23, 2013 – SQLstream Inc., the streaming Big Data engine for real-time operational intelligence and a Silver level member in Oracle… Read more →

September 24, 2013 by in Press Releases + ,

Distinguished analyst Robin Bloor has long been discussing the term of “data science,” with posts ranging from explanatory to complaining (particularly about the use of terminology).
Throughout all, however, he recognizes data science as a sensible activity for a business to pursue.
In his last post, he is promoting that idea,… Read more →

SQLstream Explores the Total Cost of Streaming Big Data Solutions at TDWI Solution Summit 2013, September 15-17 — CEO Damian Black to Present “Operational Intelligence from Machine Big Data: Case Studies in Taming the Cost of Real-time Performance”
San Francisco, CA, September 4, 2013 — SQLstream Inc., the streaming Big… Read more →

Streaming Hadoop HBase Case Study: Turbo-charging Hadoop for real-time machine data analytics. Session to be Presented at NoSQL Now! on August 22nd at 9:30 PST, San Jose.
San Francisco, CA | August 19, 2013 – SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today… Read more →

Ventana’s Technology Innovation Awards showcase “advances in technology that contribute significantly to improved efficiency, productivity and performance of the organization.” SQLstream’s IT Analytics and Performance award recognized SQLstream’s innovative technology and ability to optimize operational processes and systems. The award considered all aspects of SQLstreams technology and business approach,… Read more →

Damian Black, SQLstream CEO, is interviewed by Dataversity prior to his presentation at NoSQL Now! 2013. Speaking on streaming SQL and operational intelligence for Hadoop and HBase, Damian discusses the challenges of accelerating Hadoop to process live, high velocity unstructured data streams, and the emergence of SQL as the… Read more →

August 14, 2013 by in In the News

The definition of machine data covers, not surprisingly, all data generated by machines – servers, applications, sensors, web feeds, networks and service platforms. It covers everything from data centers, telecommunications networks and services to machine-to-machine and the Internet of Things in a device-connected world.
The value of machine data is… Read more →

The number of articles about the Internet of Things [IoT], Machine-to-Machine communication [M2M], the Industrial Internet, the Internet of Everything [IoE] and the like have been increasing since this post introducing an IoT mind map almost a year ago. We all learn from some of them, with some we nod sagely in agreement,… Read more →

August 1, 2013 by in In the News + , ,

Folklore has it that the term ‘Internet of Things’ (IoT) was first popularized in 1999 at MIT to describe the architecture of connected RFID devices. Cisco then looked to define when the IoT came in to being as a concrete entity – defined as the year in which the… Read more →

The recent public release of information on the activities of our Intelligence Agencies makes interesting reading for Big Data professionals. In particular, how these agencies have mastered the Internet, with vast data collection and analytics facilities for monitoring Internet and smartphone activity. A series of news articles described GCHQ’s… Read more →

July 5, 2013 by in Market Views

Webinar Series to Explore Challenges and Solutions for Low-latency Operational Intelligence in the Telecommunications, Cybersecurity and Large-scale Sensor Network Industries
San Francisco, CA | June 27,2013 – SQLstream, Inc., the leading standards-based Streaming Big Data Platform, announced today that it will be hosting a series of educational webinars to take place… Read more →

We participated on the “Architecting Big Data Systems for Speed” panel at E2 Conference. Great event, and a great opportunity to discuss technology in a business context. The panel offered a range of perspectives with other panelists from Translattice and Oracle’s NoSQL division. A number of interesting topics emerged,… Read more →

June 19, 2013 by in Streaming Analytics
GigaOM: Structure 2013 Logo

“The Total Cost of Performance in a Massively Connected World” Session to be Presented at GigaOM Structure on June 19th at 4:55PM PST in San Francisco
San Francisco, CA | June 18, 2013 –
SQLstream Inc., a pioneer of the streaming Big Data engine for real-time operational intelligence, today announced that SQLstream… Read more →

UBM TECH 32 CONFERENCE

Ronnie Beggs Participating on “Big Data: Architecting Systems at Speed ” Panel at E2 Conference, June 18th at 2:30PM ET, Boston
San Francisco, CA | June 17, 2013 – SQLstream Inc., the streaming Big Data platform, today announced that SQLstream has been invited to speak at E2 Conference on the architectural… Read more →

Sensors Expo is the leading industry event for the types of intelligent sensor-integrated systems that are driving the next generation Internet of Everything, Industrial Internet, telematics and Machine-to-Machine services. As a key sponsor in the Big Data and Wireless Systems pavilion, and speaking in the Big Data track, we found… Read more →

SQLstream was named as one of the top 100 companies that provide technologies and services for data management by the Database Trends and Applications (DBTA) magazine.
A result of lengthy research and compiled by the publication’s editorial staff, the first “DBTA 100: The Companies That Matter Most in Data” list recognizes industry… Read more →

InfoArmor and SQLstream Team Up to Deliver the Next Generation of Continuous Identity  Monitoring Services at Massive Scale 
SAN FRANCISCO – JUNE 5TH, 2013 – SQLstream, Inc., the Streaming Big Data Company, today announced that InfoArmor, a U.S. leader in identity theft monitoring and internet surveillance, selected SQLstream as the… Read more →

logo_sensors_0

San Francisco, CA | May 30, 2013 – SQLstream Inc., the Streaming Big Data Company, announced today that SQLstream VP Americas, Glenn Hout, has been invited to speak on real-time operational intelligence and prescriptive analytics for the Internet of Everything at Sensors Expo 2013, Jun 4-6.
Held in Rosemont, Illinois,… Read more →

The Internet of Everything is the new frontier for real-time and Big Data, where Velocity now trumps Volume as the primary driver, where the geographical distribution of streaming data adds new levels of complexity, and yet the useful lifetime of data, the window within which to make a decision,… Read more →

Real-time to Action for Stream Processing

SQLstream sponsored the recent IE Group Big Data Innovation Summit in San Francisco where I also presented on streaming SQL for Hadoop, and extending Hadoop for real-time operational intelligence and streaming analytics. As Big Data technologies and Hadoop push further into mainstream enterprises, so the need for real-time business… Read more →

300x200-Blog-BigData

Unlike traditional relational database management systems (RDBMS), which persist data that can later be queried, SQLstream runs continuous queries on targeted data streams – enabling a real-time view of an organization’s situation.
Eric Kavanagh recaps SQLstream’s last session in The Briefing Room, held alongside analysts Mark Madsen and Robin Bloor-… Read more →

April 9, 2013 by in In the News

San Francisco, CA | February 20, 2013 – SQLstream, the provider of the real-time data hub for stream processing applications, today announced an extension to its partner program that helps to fulfill the growing need for real-time data processing systems in supercomputing centers and academic cyberinfrastructure facilities. The SQLstream… Read more →

February 20, 2013 by in Press Releases
300x200-Blog-BigData

Today’s edition of Information Management’s DM Radio Broadcast, The Future of Integration: ETL, CDC and IOA, had a great panel line up discussing the breadth of data integration issues in today’s world of Big Data, Cloud and traditional enterprise architectures. The session was hosted by Eric Kavanagh (Bloor) and… Read more →

300x200-Blog-BigData

Bloor Group’s Robin Bloor hosted SQLstream’s CEO Damian in The Briefing Room in a webcast entitled “Windows of Opportunity: Big Data on Tap”. The webinar focussed on the emergence of both SQL and the stream processing as a key enabler for real-time Big Data systems in an ever-maturing marketplace…. Read more →

300x200-Blog-Feature-News

Dylan Janus from Information Today talks about the latest SQLstream release on DTBA, and how SQLstream s-Server provides faster response to operational Big Data. The flagship SQL stream processing platform introduces high performance distributed stream processing, with Google BigQuery integration, enhanced platform manageability and enhanced support for streaming analytics…. Read more →

300x200-Blog-BigData

The SQLstream Briefing Room webinar with Robin Bloor “Windows of Opportunity: Big Data on Tap”  highlighted how the evolving Big Data landscape needs technologies that enable a much bigger enterprise-wide picture, complete with multiple data streams that can be combined to show what’s happening in real-time. The speakers were… Read more →

I have recently researched IBM® Infosphere Streams’ “Stream Processing Language” or SPL, after a pretty good talk at a recent SVForum SIG group meeting. Early users of this technology (which are really PS projects disguised as license sales) found that their Version 2.0 Stream Processing Application Declarative Engine (SPADE) programming language has… Read more →

December 20, 2012 by in Market Views

New SQLstream s-Server 3.0 enables faster response to operational Big Data by tapping into streaming log file, sensor and service data in real-time.
SAN FRANCISCO, CA, December 13, 2012 – SQLstream Inc. (www.sqlstream.com), a pioneer of real-time Big Data, today delivered the new generation of its streaming Big Data management… Read more →

Contact Us

Would you like to know more about stream processing?

+1 877 571 5775

Ask a Question