Skip to content
Try for free
Try for free

How to update stock market data automatically with CrateDB and Airflow

In this video, I show how to update stock market data automatically with CrateDB and Airflow.
First, I show how to run CrateDB with Docker and create a table to store the financial data. Then, to orchestrate the process of regular data updates, I will make an Airflow project and establish the connection to CrateDB. Once I set up the Airflow project, I show how to write workflow tasks and dependencies in Python as nodes and edges of an Airflow DAG. With the ready DAG, I execute the tasks in the Airflow UI and show the data in CrateDB.
Follow this tutorial in written form at

Overview - 00:28
Run CrateDB and create a table - 01:38
Install Astronomer and initialize Airflow project - 03:40
What is a DAG? - 07:15
Import operators and modules - 08:11
Download task - 09:24
Prepare data task - 11:21
Insert data task - 12:08
Set task dependencies - 12:58
Put DAG together - 14:06
Execute DAG in Airflow UI - 14:50
Check out data in CrateDB - 16:03


Useful links:

Presenter: Rafaela Sant'ana, Customer Engineering