No enterprise wants malicious intruders hacking their way into mission-critical systems. Nevertheless, Web or VPN access to information is critical for many employees, business partners, and customers. If people outside the corporate firewall can tunnel their way through, then there is always the chance that hackers can, too.

Standard Response

Enterprises often respond to this ever-present danger by installing Security Event Monitoring applications. This SEM software continuously monitors log file streams of server data recording attempts by applications or users to log onto systems through web or dial-up access. Such attempted access can come from routers, firewalls, web proxies, web servers, and many other applications and systems. SEM software normally works by collecting, cleaning, and aggregating the data collected from such sources to create more complete user session records, ultimately inserting them into a database.

In the database system, queries run continually, checking every few minutes for tell-tale signatures of hackers or other malicious users. Suspicious patterns include repeated attempts to crack passwords, or perhaps to hit a number of Internet addresses or ports in sequence, looking for an unprotected address/port combination.

Difficulties with Standard Response

The more sophisticated the queries, the less frequently they can be executed, because database systems cannot efficiently process high-volumes of input record insertions concurrent with elaborate queries.

For example, a common transaction rate of well over thirty thousand insertions per second is now common for SEM work. This high a rate requires very specialized database technologies developed specifically for this domain, because such an insertion rate outstrips the capabilities of conventional RDBMS. The problem will only get more acute with network bandwidth increases and with the explosively rising number of systems potentially connecting to an enterprise's computer systems.

The need for such high performance insertions while repeatedly running a number of important queries has given rise to a specialized product niche called "high insertion rate" databases. Mainstream RDBMS are simply not capable in a cost effective manner of keeping up with the workload.

In addition to looking for suspicious access patterns, SEM applications also produce rolling reports such as "top ten lists," providing indicators of potential attack attempts - both past and present. Examples include lists of the most active users, the most unsuccessful log-in attempts per IP address, and so on.

SEM Requirements and SQLstream

Without SQLstream, other solutions have difficulty dealing with the following logical questions:

• Why ship all of the data to a central RDBMS where the data cannot be loaded fast enough except by using specialized high-insert rate database?

• Why not run the queries to identify intruder patterns as the data are transmitted over the network?

SQLstream is ideally suited to SEM requirements - seeking suspicious access patterns and calculating rolling top ten lists - because the RAM model enables distributing all such queries out onto the network. They remain continuously active and process the data at very high throughput rates.

SQLstream can process the messages without writing them transactionally to disk, without updating disk base indexes, and without rerunning or restarting the same queries again and again for each new input record.

Consider the update/query cycle prevalent in SEM applications using their "high insertion rate" databases, as illustrated in this drawing:

Due to these prohibitive expenses incurred by running a set of queries on every insertion, SEM applications today only run the queries every N seconds for some configured parameter. SQLstream, on the other hand, can happily crunch through all of the queries for every message, without those prohibitive expenses.

Using SQLstream in SEM applications complements the role of a high-performance database, because SQLstream RAMMS provides the following capabilities:

• Supplies a much more cost-effective way to do continuous SEM queries, with every logged transaction message accounted for

• Writes summary records and inserts such summary records into an external database, with the option of allowing all of the original input records to flow through to other queries or processes

• Enables ad hoc queries to examine past activities of an intruder once detected

• Significantly offloads queries otherwise sent to a high-performance database if one is in use

• Enables the use of more mainstream, cheaper, or more powerful querying database technologies rather than specialized high-insertion rate technologies

• Allows higher insertion-rates (transaction rates)

• Enables near-instantaneous detection of intruders, because it can execute all queries against devery message as it arrives

• Makes the collected data simultaneously available to other applications that are running concurrently, such as performance monitoring for networks or applications, Service Level Agreement (SLA) conformance monitoring, and applications for charging and billing