Database for Log Analysis
that truly Scales
Log analysis consists in reviewing computer-generated event logs to proactively identify bugs, security threats or other risks. Log analysis can also be used to review user behavior or ensure compliance with regulations.
CrateDB is a database perfectly suited to log analysis thanks to the combination of a real-time SQL engine, a NoSQL foundation, and a full-text search engine. It offers the scalability, performance, and flexibility to process any type of logs and massive volumes of logs in real-time, supporting a wide variety of log analytics use cases, such as cybersecurity, network performance monitoring, video analytics.
Millions of data points per second

Intuitive data modeling
CrateDB can collect any type of data (structured, unstructured, semi-structured, time-series, geospatial, BLOB) and can accept any type of log without any need for specific development.

Immediate data indexation and all data fields indexed

Native SQL

Real-time query performance

Always on

Interested?
Examples of log analysis queries
/*
* Based on system event logs, this query calculates:
* - a filter for specific messages using a full-text index
* - the number of entries per minute
* - the average scoring ratio for each matched row
*/
SELECT DATE_TRUNC('minute', receivedat) AS event_time,
COUNT(*) AS entries,
AVG(_score) AS avg_score
FROM "syslog"."systemevents"
WHERE MATCH(message, 'authentication failure')
USING most_fields WITH (analyzer = 'whitespace')
AND MATCH(syslogtag, 'sshd')
GROUP BY 1
ORDER BY 1 DESC
LIMIT 10;
+---------------+---------+--------------------+
| event_time | entries | avg_score |
+---------------+---------+--------------------+
| 1620220260000 | 4 | 1.5798743814229965 |
| 1620220200000 | 8 | 1.7750384211540222 |
| 1620220140000 | 10 | 1.6113891124725341 |
| 1620220080000 | 9 | 1.676726798216502 |
| 1620220020000 | 8 | 1.6908064410090446 |
| 1620219960000 | 8 | 1.690401442348957 |
| 1620219900000 | 7 | 1.7646006005150932 |
| 1620219840000 | 7 | 1.7795820917401994 |
| 1620219780000 | 10 | 1.5844267368316651 |
| 1620219720000 | 13 | 1.5637413492569556 |
+---------------+---------+--------------------+

"It is through the use of CrateDB that we are able to offer our large-scale video analytics component in the first place. Comparable products are either not capable of handling the large flood of data or they are simply too expensive."
Daniel Hölbling-Inzko
Senior Director of Engineering
Bitmovin



Other log analysis resources
Tutorial
Storing server logs on CrateDB for fast search and aggregations
Tutorial
Using regex comparisons and other advanced database features for real-time inspection of web server logs
There are a lot of nice features in CrateDB to support this kind of setup. Learn how.