Consistency & Durability

CrateDB adopts an eventual consistency model for search operations, optimized for performance and availability in distributed systems.

This model is built around shared IndexReaders, which cache shard data and enable rapid lookups and aggregations.

• Each IndexReader is bound to a Lucene segment and refreshed periodically to incorporate recent changes.
• Refreshes can be time-based (configurable) or triggered manually via the REFRESH TABLE command.
• A search query only “sees” data that’s been indexed and refreshed, ensuring predictable, low-latency search behavior.

This design provides the best of both worlds: real-time analytics with continuous availability, even during heavy ingestion or partial network outages.

For queries where the primary key or routing key is fully specified (for example, WHERE id = '123'), CrateDB enforces strong consistency.

• The system can directly compute the shard and row location.
• The document is retrieved through a single, atomic lookup on the _id field.

This guarantees that reads and writes on the same row are immediately consistent, a crucial property for transactional operations, upserts, and precise updates.

While CrateDB is designed for horizontal scalability and high ingestion rates, it never compromises on data integrity.
Every write operation in CrateDB is atomic at the row level:

• An insert, update, or delete either fully succeeds or has no effect at all.
• Even complex or nested documents are treated as single atomic entities.
• No partial writes, no corrupted states.

Durability is ensured through write-ahead logging and shard replication: once a write is acknowledged, the data is persisted and replicated to ensure recovery even if a node fails.

Combined, these mechanisms safeguard the integrity of your data across distributed clusters, even under extreme concurrency or hardware interruptions.

CrateDB leverages Multi-Version Concurrency Control (MVCC) to handle concurrent operations efficiently.

• Multiple read and write transactions can occur simultaneously without locking contention.
• Readers always see a consistent snapshot of the data.
• Writers can update rows independently without blocking queries.

MVCC ensures that CrateDB maintains both real-time performance and consistency, allowing analytical and operational queries to run smoothly in parallel.

CrateDB’s consistency and durability model is designed for modern data workloads:

• Real-time analytics on continuously changing data.
• Operational monitoring with guaranteed correctness on key lookups.
• IoT and manufacturing data where temporary network partitions cannot halt processing.
• AI and ML pipelines where feature data must remain accurate and recoverable.