Load and Export

You have a variety of options to connect and integrate with 3rd-party ETL applications, mostly using CrateDB’s PostgreSQL interface.

This documentation section lists corresponding ETL applications and frameworks which can be used together with CrateDB, and outlines how to use them optimally. Please also have a look at support for Change Data Capture (CDC) solutions.

Apache Airflow / Astronomer

A set of starter tutorials.

• Automating the import of Parquet files with Apache Airflow

• Updating stock market data automatically with CrateDB and Apache Airflow

• Automating stock data collection and storage with CrateDB and Apache Airflow

A set of elaborated tutorials, including blueprint implementations.

• Automating export of CrateDB data to S3 using Apache Airflow

• Implementing a data retention policy in CrateDB using Apache Airflow

• CrateDB and Apache Airflow: Building a data ingestion pipeline

• Building a hot and cold storage data retention policy in CrateDB with Apache Airflow

Tutorials and resources about configuring the managed variants, Astro and CrateDB Cloud.

• ETL with Astro and CrateDB Cloud in 30min - fully up in the cloud

• ETL pipeline using Apache Airflow with CrateDB (Source)

• Run an ETL pipeline with CrateDB and data quality checks

Apache Flink

• Data Ingestion using Kafka and Kafka Connect

• Build a data ingestion pipeline using Kafka, Flink, and CrateDB

• Community Day: Stream processing with Apache Flink and CrateDB

• [Executable stack: Apache Kafka, Apache Flink, and CrateDB]

Apache Hop

• Using Apache Hop with CrateDB

• CrateDB dialect for Apache Hop

• CrateDB Apache Hop Bulk Loader transform

Apache Kafka

• Data Ingestion using Kafka and Kafka Connect

• Executable stack with Apache Kafka, Apache Flink, and CrateDB

• Replicating data to CrateDB with Debezium and Kafka

Apache NiFi

• Connecting to CrateDB from Apache NiFi

Azure Functions

• Data Enrichment using IoT Hubs, Azure Functions and CrateDB

dbt

• Using dbt with CrateDB

InfluxDB

• Import data from InfluxDB

Kestra

• Setting up data pipelines with CrateDB and Kestra

MongoDB

• Import data from MongoDB

MySQL

• Import data from MySQL

Node-RED

• Ingesting MQTT messages into CrateDB using Node-RED

• Automating recurrent CrateDB queries using Node-RED

Singer / Meltano

• meltano-target-cratedb

• meltano-tap-cratedb

• Examples about working with CrateDB and Meltano

🚧 Please note these adapters are a work in progress. 🚧

SQL Server Integration Services

A demo project which uses SSIS and ODBC to read and write data from CrateDB:

• Using SQL Server Integration Services with CrateDB

StreamSets

• Data Stream Pipelines with CrateDB and StreamSets Data Collector