How to update stock market data automatically with CrateDB and Airflow

In this video, I show how to update stock market data automatically with CrateDB and Airflow.
First, I show how to run CrateDB with Docker and create a table to store the financial data. Then, to orchestrate the process of regular data updates, I will make an Airflow project and establish the connection to CrateDB. Once I set up the Airflow project, I show how to write workflow tasks and dependencies in Python as nodes and edges of an Airflow DAG. With the ready DAG, I execute the tasks in the Airflow UI and show the data in CrateDB.
Follow this tutorial in written form at https://community.crate.io/t/how-to-update-stock-market-data-automatically-with-cratedb-and-airflow/1304

Chapters:
Overview - 00:28
Run CrateDB and create a table - 01:38
Install Astronomer and initialize Airflow project - 03:40
What is a DAG? - 07:15
Import operators and modules - 08:11
Download task - 09:24
Prepare data task - 11:21
Insert data task - 12:08
Set task dependencies - 12:58
Put DAG together - 14:06
Execute DAG in Airflow UI - 14:50
Check out data in CrateDB - 16:03

Useful links:

• https://brew.sh
• https://www.docker.com
• https://community.crate.io/t/cratedb-and-apache-airflow-automating-data-export-to-s3/901
• https://www.astronomer.io/docs/astro
• https://www.astronomer.io/docs/astro/cli/install-cli
• CrateDB and Airflow repository https://github.com/crate/crate-airflow-tutorial
• More content on CrateDB and Airflow:
  • https://crate.io/resources/videos/dynamic-workflow-orchestration-with-apache-airflow-and-cratedb
  • https://community.crate.io/t/cratedb-and-apache-airflow-implementation-of-a-data-retention-policy/913
  • https://crate.io/blog/check-the-quality-of-your-data-using-dynamic-orchestration-with-airflow

Presenter: Rafaela Sant'ana, Customer Engineering Crate.io