We’re at Sensors Expo this week, showcasing in the Big Data & Analytics Pavilion. This is the first year the event has included a specific area for real-time Big Data solutions for sensor networks.
Real-time control in a Big Data World
SQLstream CEO, Damian Black, presented on Real-time Control in a Big Data World. The presentation focused on the increase of sensor data and the emergence of the “Sensor Internet”, plus the applications required to collect and analyze streaming sensor data, and to drive real-time actions and updates. In particular, addressing the emerging real-time Big Data challenges in this area driven by wireless and GPS technologies, M2M applications and V2V/V2I.
Real-time streaming data integration for Big Data
It’s clear the primary challenge is not managing the data volume per se, or even delivering real-time operational intelligence, rather it’s the more fundamental issue of real-time streaming data integration. How can such huge volumes of data from many different sources and locations be integrated into the operational platforms, and how can the issues of multiple operational siloes be overcome to provide an integrated real-time control platform.
Interestingly, these are the exact same issues SQLstream addresses for the Big Data and Hadoop world in general – getting data in, getting data out, connecting existing data stores in real-time, and delivering real-time in-memory analytics on the data as it streams past:
- Real-time streaming data integration of any data source and between existing storage platforms and operational systems
- Real-time streaming monitoring and analytics on the arriving and streaming data
- Scalability through parallel distributed processing of processing pipelines
The importance of geospatial analytics
Geospatial analytics is a key requirement in the sensor data market. Big Data analytics in general is about one dimensional problems, usually the correlation of similar events, or the correlation of events over time. The geospatial dimension is the key difference between Big Data platforms for the “Sensor Internet” and the wider IT / machine data applications. Fortunately this has been a feature of SQLstream for some time, and central to many of our customer deployments. For example, real-time traffic analytics from GPS data, and real-time seismic monitoring.