Time Series Databases and Operational Historians: Get the Best of Both Worlds With CrateDB and the Crate IoT Data Platform

If you're working with time series data, and you want to leverage business intelligence, you have two main options:

1. Develop and maintain custom business intelligence software on top of a modern time series database.
2. Deal with the hassles of running a traditional operational historian that provides most of the business intelligence you need.

Neither one is ideal.

However, there is a third option.

But, before I go on, let's take a closer look at the issues.

Time series databases

A time series database is any database that is particularly well suited for handling time series data (e.g., digital sensor readings). If you use something like Prometheus to monitor the performance of your IT systems, that is time series data in action.

However, if you want to use a time series database for business intelligence, you are going to have to develop and maintain your own software for that. For many businesses, that is too costly. And so, even when an organization is producing time series data, a lot of its potential is left unrealized.

Also, the scale of the Industrial Internet of Things (IIoT) can present a challenge for some of these products.

Connected factories, energy networks, smart city infrastructure, and vehicle fleets, and so on, are capable of generating workloads that are orders of magnitude bigger and more complex than traditional time series use-cases. And with that, comes an increased need for ingestion, storage, and processing capacity.

Many businesses turn to a type of software application called an operational historian to address these concerns.

Operational historians

An operational historian combines the functionality of a time series database with turnkey business intelligence solutions designed to help a business to leverage the value of operational process data (e.g., data collected from a manufacturing plant's instrumentation and control systems).

Compared with time series databases, the typical benefits of a historian include:

• Handle orders of magnitude more data
• Ready-to-use connectors for a wide range of different input sources
• Aggregation, data cleansing, and data enrichment
• Built-in quality control, anomaly detection, and process management systems
• Business intelligence, analytics, and reporting
• Fine-grained access controls

These benefits are so compelling that, according to DiscoverOrg, more than 74% of Fortune 500 manufacturing companies run historians.

However, drawbacks also include:

• The historian market is dominated by decades-old legacy time series platforms that don't take advantage of modern engineering techniques and weren't designed for today's technology landscape.
• Historians are costly to own, difficult to scale, and your range of deployment options is usually limited.
• Modern architectural design and real-time analytics requirements can not be realized, as many historians use dated or proprietary communication protocols and sometimes require specialized client applications—making them difficult or impossible to integrate with other systems and tools.
• Few historians offer real-time, actionable insight at massive scale. Instead, they tend to be used as proverbial rearview mirrors.

So, it appears that businesses with a tough decision to make.

Should you roll your own historian by writing custom software on top of a modern time series database?

Or do you want to go with a legacy stack that comes with batteries included, even if that means having to deal with the drawbacks above?

The best of both worlds

With CrateDB and the Crate IoT Data Platform, you can get the best of both worlds: a modern time series database that comes with batteries included (i.e., can do everything a traditional operational historian can do).

A modern time series database

CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time.

Because CrateDB is designed using modern technology, it can handle industrial-scale time series data workloads with ease:

• Ingest, store, and process millions of data points per second thanks to CrateDB's data partitioning, multithreaded design, and shared-nothing distributed architecture with masterless clustering that allows you to grow to an unlimited size.
• Get the power of ANSI SQL (including interoperability and no lock-in) without sacrificing the elastic scalability associated with NoSQL databases and public cloud Databases as a Service (DBaaS).
• Deploy CrateDB with container tools like Docker and Kubernetes and scale up or down with a single command to linearly increase or decrease capacity.
• Real-time query performance optimized for time series data, even at scale—with the help of in-memory columnar indexes, field caches, and a distributed query execution engine that can parallelize complex queries across the whole cluster.
• Built-in features for high-availability and disaster recovery—including sharding, replication, self-healing, auto-rebalancing, and snapshots.

Batteries included

Using the modern technology of CrateDB as a foundation, Crate.io has built a lighthouse vertical for discrete manufacturers: the Crate IoT Data Platform.

The Crate IoT Data Platform is a fully-managed operational historian and mission control Platform as a Service (PaaS) that runs on Microsoft Azure.

With The Crate IoT Data Platform, you can:

• Take advantage of the scalability, security, and cost-effectiveness of the cloud without having to build your own cloud-native deployment stack.
• Connect all of the devices and sensors across all of your factories without having to sacrifice the benefits of real-time data collection, enrichment, monitoring, analysis, and querying.
• Leverage machine learning (e.g., predictive maintenance) and run time series operations, geospatial queries, and fulltext search at any scale.
• Take action on business intelligence "in-the-moment." Make course corrections in seconds instead of days.
• Add your own software and tools into the mix. CrateDB’s standard SQL interface allows you to customize and extend the platform as much or as little as you wish.

The proof is in the pudding

We have customers today who are using the Crate IoT Data Platform to:

• Improve factory efficiency (OEE)
• Raise profitability
• Reduce waste
• Increase safety and security

For example, ALPLA is a $4B global plastic packaging manufacturer with almost 2,000 production lines in 180 manufacturing sites across 45 countries. They produce millions of products per day, and their goal is to optimize efficiency by saving pennies on the dollar in their high-speed production lines.

We helped ALPLA build a mission control center using CrateDB and the Crate IoT Data Platform that connects tens of thousands of sensors and provides real-time monitoring and alerting, predictive maintenance, and augmented reality connected to the factory floor.

The mission control center was rolled out to 18 plants in 2019, and ALPLA saw seven-figure savings through efficiency improvements, raw material waste savings, and maintenance cost reduction.

Another $B manufacturer is using CrateDB to reduce the costs of poor quality (COPQ). For them, CrateDB provides 10x faster ingest and is 30% less expensive than their previous solution. What's more, they get to use one database for both stream data and historian data.