Time Series Database for Real-Time Analytics

Sensors, applications, and infrastructure emit measurements continuously. A time series database accepts thousands to millions of writes per second without index maintenance overhead slowing the ingestion path.

Queries almost always include a time range filter, a time-based grouping (per minute, per hour, per day), or a rolling window calculation. A time series database makes these operations fast at any data volume. 

A real-world deployment might include millions of unique combinations of device ID, region, sensor type, and status code. High cardinality is not an edge case, it is the norm. A time series database handles it without performance degradation.

Time series data grows indefinitely. A time series database manages hot, warm, and cold data tiers, enforces retention policies, and downsizes old data without manual intervention.

Industrial facilities generate continuous streams of sensor data from equipment, production lines, and environmental systems. TGW, a global logistics automation company, uses CrateDB to process data from 900,000 sensors across a single distribution center. The data (temperature, vibration, power draw, throughput) flows in continuously and must be queryable in real time to support equipment monitoring, anomaly detection, and operational dashboards.

At this sensor density, a database that cannot sustain the ingestion rate without backpressure or data loss is not viable. CrateDB handles the volume while keeping analytical queries fast enough for live dashboard refresh rates.

High-frequency sensor data is the input for any predictive maintenance model. To detect an emerging failure before it causes downtime, the database must ingest measurements at high frequency (often sub-second intervals), retain the full history without sampling, and serve both live dashboards and ML inference pipelines from the same store.

ABB, the industrial technology company, processes 1 million sensor values per second in CrateDB for this workload. The same database that accepts the ingestion stream answers the analytical queries that feed the maintenance model.

Application performance monitoring generates time series data at high cardinality: every combination of service, host, region, endpoint, and status code is a unique time series. Systems that cannot handle cardinality at scale drop data, aggregate too aggressively, or become expensive to run past a certain point.

Upstream and downstream operations generate continuous streams of sensor and operational data: wellhead pressure, flow rates, pipeline conditions, and equipment status across distributed assets that may span hundreds of kilometers. SPGO processes 750 million records per day in CrateDB, querying operational data across their asset network while it is still live.

Video platforms and content delivery networks generate event streams at massive scale. Bitmovin, a video technology company, uses CrateDB to analyze 60 million rows in under a second, supporting real-time quality-of-experience analytics across global viewer sessions. The queries run over raw event data, without pre-aggregated summaries.

Relational databases update indexes on every insert. Under continuous high-frequency ingestion, index maintenance consumes CPU and I/O faster than the database can process incoming data. Write throughput collapses. The common fix consisting in disabling indexes then breaks query performance.

A SELECT with a WHERE timestamp > X AND timestamp < Y clause requires a full table scan without a time-optimized index structure. As the table grows to billions of rows, scan times grow with it. Pre-aggregation tables become mandatory, which introduces staleness and a maintenance burden that grows with the number of required aggregations.

When every combination of device ID, region, and metric name is a unique series, label tables grow into hundreds of millions of rows. Queries that join against these labels become slow. Most databases either force you to limit cardinality or degrade silently as it grows.

New sensor types, new metadata fields, and new measurement types require schema migrations. In production environments with continuous ingestion, migrations are painful and often avoided, which accumulates schema debt that eventually requires a rewrite.

Managing partitioning, archiving, retention policies, and replication on a general-purpose database requires custom engineering for every new scale tier. The cost is not in the database license, it is in the engineering time spent working around a tool that was not designed for this workload.

The first question is whether the database can accept your write volume without backpressure, dropped data, or write-induced query degradation. Some databases queue writes behind index maintenance. Others partition writes across nodes but bottleneck on the coordinator. Test ingestion under your expected peak load — not average load — before committing.

For industrial and IoT workloads, peak ingestion during plant startup, incident response, or event bursts often exceeds average rates by a factor of 5 to 10.

Cardinality is the number of unique time series in your dataset. In a sensor deployment, it is roughly the number of distinct combinations of device ID, metric name, and metadata tags. Some databases store cardinality in memory and impose hard limits. Others degrade as cardinality grows past a certain point without surfacing a clear error.

If your deployment has thousands of devices today and could have tens of thousands in two years, test with realistic projected cardinality, not current cardinality.

Most specialized time series databases use proprietary query languages: InfluxQL, Flux, and Prometheus PromQL each require dedicated expertise and do not integrate natively with SQL-based BI tools, notebooks, or data engineering frameworks.

A database with standard SQL support means every SQL-literate engineer on your team can query time series data immediately. It also means your existing tooling (dashboards, ETL pipelines, notebooks) connects without a custom adapter. For teams that need to join time series data against relational metadata (asset registry, user data, reference tables), SQL is not optional.

Time series data accumulates indefinitely. A database that stores everything at full resolution with no tiering strategy will become expensive to operate past a certain scale. Evaluate how the database handles:

• Automated retention policies: does it drop old data on a schedule without manual intervention?
• Tiered storage: can it move older data to cheaper storage while keeping it queryable?
• Downsampling: does downsampling happen automatically, or does it require a separate aggregation pipeline?
• Some workloads: compliance, root-cause analysis, ML training require full-resolution historical data for years. Choose a database that can retain it without requiring a separate archival system.

New sensor types, new metadata fields, and new measurement categories arrive continuously in real deployments. A database that requires a migration for every schema change creates friction at exactly the moment you need to move fast.

Evaluate how the database handles: adding new columns without downtime, storing heterogeneous payloads from devices with different schemas, and querying fields that exist only in a subset of records.