Edit on github

Creating Aiflow Dags

Pre-Requisites

By now you should have:

Where to create your DAGs

This means that Airflow is fully configured and we can turn our attention to creating DAGs! Airflow uses DAGs to run dbt as well as other orchestration tasks. Below are the important things to know when creating DAGs and running dbt with Airflow.

During the Airflow configuration step you added the orchestrate folder and the dags folder to your repository. Here you will store your airflow DAGs. ie) You will be writing your python files in orchestrate/dags

DAG 101 in Datacoves

  1. If you are eager to see Airflow and dbt in action within Datacoves, here is the simplest way to run dbt with Airflow.

    Run dbt

  2. You have 2 options when it comes to writing DAGs in Datacoves. You can write them out using Python and place them in the orchestrate/dags directory, or you can generate your DAGs with dbt-coves from a YML definition.

    Generate DAGs from yml definitions this is simpler for users not accustomed to using Python

  3. You may also wish to use external libraries in your DAGs such as Pandas. In order to do that effectively, you can create custom Python scripts in a separate directory such as orchestrate/python_scripts and use the DatacovesBashOperator to handle all the behind the scenes work as well as run your custom script. You will need to contact us beforehand to pre-configure any python libraries you need.

    External Python DAG