Data Storage
CrateDB provides an efficient data storage solution by employing both sharding and optional partitioning for every table. This results in tables being divided and strategically distributed across the nodes of a cluster. A unique attribute of CrateDB is that each shard corresponds to a Lucene index, which is comprised of segments stored directly on the file system. The physical files, in turn, are located under one of the configured data directories of a node.
Lucene engine's characteristic of only appending data to segment files ensures that data written to the disk is never mutated. This feature simplifies replication and recovery processes as syncing a shard becomes a simple task of fetching data from a specific marker.
As segments grow over time, CrateDB automatically performs periodic merges. During a merge, redundant documents marked as deleted are discarded. The new segments generated post the merge contain only valid, non-deleted documents from the initial segments. Users can also initiate a segment merge using the OPTIMIZE TABLE command. Another important feature to note is CrateDB's use of 'soft delete'. This accelerates the replication and recovery process. The period for which deleted documents need to be preserved to free up disk space is determined by the retention lease period. Documents are discarded only after the expiration of this period, with the default being set to 12 hours.
- Partitioning and sharding: CrateDB employs partitioning and sharding to improve data management and query performance. This involves dividing tables into partitions and further into shards, which are then distributed across the cluster.
- Replication: Each table shard in CrateDB has configured copies to enhance fault tolerance and read performance. The replication process involves synchronizing data from the primary shard to all replica shards.
- Atomicity, durability, and consistency: CrateDB enforces these principles to ensure the integrity of data operations. Atomicity is applied at the document level, durability is ensured through a Write Ahead Log (WAL), and consistency levels are maintained for 'get' and 'search' queries.
- Advanced indexing: To facilitate efficient data retrieval, CrateDB automatically indexes every attribute by default, utilizing strategies like Inverted Index for text values, Block k-d trees for numeric, date, and geospatial values, and Hierarchical Navigable Small World (HNSW) graphs for high dimensional vectors.
- Columnar storage: CrateDB adopts a columnar storage approach to optimize query performance and complex aggregations across large datasets. This approach optimizes file-system cache use and allows certain queries to skip unnecessary blocks during execution.
- Data tiering: CrateDB utilizes data tiering to manage data across different storage tiers based on access frequency and performance needs. This results in cost-effective data management and optimized resource utilization.
CrateDB Architecture Guide
This comprehensive guide covers all the key concepts you need to know about CrateDB's architecture. It will help you gain a deeper understanding of what makes it performant, scalable, flexible and easy to use. Armed with this knowledge, you will be better equipped to make informed decisions about when to leverage CrateDB for your data projects.
