📄️ What to know
Important notes before writing Airflow DAGs in Datacoves: Ruff linting, Datacoves decorators, and My Airflow for faster personal DAG development.
📄️ Initial Setup
Configure Airflow in Datacoves: set DAG paths, git sync branch, Kubernetes executor settings, and environment-level service connections for dbt jobs.
📄️ Airflow REST API
Authenticate with and call the Airflow REST API in Datacoves: generate API tokens, trigger DAGs, check run status, and query task instances programmatically.
📄️ Sync Internal Database
How to sync the internal Airflow metadata database in Datacoves to apply configuration changes, reset state, or recover from database inconsistencies.
📄️ Trigger DAG via Datasets
Step-by-step guide to triggering Airflow DAGs via dataset events in Datacoves, enabling data-driven pipeline orchestration without time-based schedules.
📄️ Key-Pair Authentication
Configure Snowflake key-pair authentication for Airflow connections in Datacoves instead of password-based credentials for improved security.
🗃️ DAGs
15 items
📄️ Notifications - Email
Configure email alerts for Airflow DAG failures in Datacoves: set up SMTP integration, define recipient lists, and handle task-level notifications.
📄️ Notifications - MS Teams
Configure Microsoft Teams notifications for Airflow DAGs in Datacoves to receive alerts when pipeline tasks succeed, fail, or are retried.
📄️ Notifications - Slack
Set up Slack alerts for Airflow DAG events in Datacoves: connect a workspace, configure channel routing, and receive success or failure messages.
📄️ Secrets - AWS Secrets Manager
Configure AWS Secrets Manager as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables.
📄️ Secrets - Azure Key Vault
Configure Azure Key Vault as the Airflow secrets backend in Datacoves to securely store and retrieve connections and variables using Managed Identity.
📄️ Secrets - Datacoves Manager
How to use the built-in Datacoves Secrets Manager to store and inject secrets into Airflow connections and variables without exposing credentials in code.
📄️ Worker - Custom Environment
Configure a custom Docker image for Airflow workers in Datacoves to include specific Python packages, libraries, or dependencies your DAGs require.
📄️ Worker - Request CPU & Memory
Configure Kubernetes resource requests for individual Airflow tasks in Datacoves to allocate specific CPU and memory limits per DAG task.