Skip to main content

How to use Datacoves Secrets Manager in Airflow

Datacoves includes a built-in Secrets Manager that allows you to securely store and manage secrets for both administrators and developers. Secrets can be stored at the project or environment level and easily shared across other tools in your stack, ensuring seamless integration and enhanced security. Creating or editing a secret in the Datacoves Secret Manager is straightforward. Be sure to prefix all secrets stored in Datacoves Secrets Manager with datacoves-.

Read variable from Datacoves Secrets Manager​

Once you save your variable in the Datacoves Secret Manager you are ready to use your variable in a DAG. This is done using Variable.get. Airflow will look in several places to find the variable.

The order of places it will look for are as follows:​

  1. AWS Secrets Manager (if configured)
  2. Datacoves Secrets Manager
  3. Airflow variables

Once a variable is found, Airflow will stop its search.

Secrets flowchart

Best practices when using a Secrets Manager variable​

  1. Always call your Variable.get from within the Datacoves Task Decorators. This ensures the variable is only fetched at runtime.
  2. Use prefixes based on where your variable is stored, like datacoves- (Datacoves Secrets Manager only searches for secrets with this prefix), aws_, or airflow_ to help identify and debug variables. For example: datacoves-mayras_secret.

Example DAG using Datacoves Secrets Manager​

from airflow.decorators import dag, task
from pendulum import datetime
from airflow.models import Variable

@dag(
    default_args={
        "start_date": datetime(2024, 1, 1),
        "owner": "Amy Chan",
        "email": "amy@example.com",
        "email_on_failure": True,
    },
    catchup=False,
    tags=["version_1"],
    description="Testing task decorators",
    schedule_interval="0 0 1 */12 *",
)
def task_decorators_example():

    @task.datacoves_bash
    def calling_vars_in_decorator() -> str:
        my_var = Variable.get("datacoves-mayras_secret")
        return "My variable is: " + my_var

    calling_vars_in_decorator()

dag = task_decorators_example()
tip

To auto mask your secret you can use secret or password in the secret name since this will set hide_sensitive_var_conn_fields to True. eg aws_mayras_password. Please see this documentation for a full list of masking words.