Deploying Apache Airflow® on Aiven Runtime

When we first introduced people to Aiven Runtime, before it even had that name, we kept getting asked: "Can we schedule things?"

If you come from a Unix background, that tends to cause a reflexive reach for the cron tool, which provides basic scheduling at fixed intervals. But if you've spent more time with distributed software, you're more likely to think of Apache Airflow®, which can do a lot more (although, yes, you can use Airflow to schedule tasks at fixed intervals if you want).

So we built a template for Apache Airflow on Aiven Runtime. This post covers what Airflow gives you, what the Airflow template for Aiven Runtime sets up, and where its limits are, so you know when to use it and when to reach for something bigger.

What does Airflow give you?

Airflow lets you describe your work as a Directed Acyclic Graph, or DAG. Think network, not an X/Y chart. A flowchart without loops.

• Tasks make up the nodes of the graph: the things to do.
• Dependencies are the edges: the links between tasks.
• Directed means each edge has one direction, from the task that runs first to the task that runs next.
• Acyclic means there are no loops. You can't have circular dependencies, so a workflow can never end up going round and round forever.

The DAG tells Airflow which tasks to run, when to run them, and in what order. You can also:

• Only follow an edge if a condition is met
• Create new tasks at runtime
• React to external data

DAGs are written in Python, and there are @dag and @task decorators to make that easier. Once you've defined your DAGs, you can visualise them in the Airflow UI, which is also provided in the Runtime deployment.

Here is a minimalistic example that runs three tasks daily:

Loading code...

As a flowchart, this looks like:

The Airflow documentation covers the details in Core Concepts / Dags. I like Airflow 101: Building Your First Workflow, which shows how Airflow can replace the classic cron job, and how you can take it beyond that.

What the template sets up

The Airflow template provides a Compose file and a Dockerfile that deploy Airflow's Standalone mode on Aiven Runtime. They:

• extend the official Apache Airflow Docker image
• run the web server, scheduler and triggerer in a single container, using airflow standalone
• use the LocalExecutor, so you don't need Redis® (or Valkey™) or Celery
• create an admin user with Airflow's Simple Auth Manager
• run database migrations automatically at startup

Airflow keeps its workflow state in a database. When you deploy with the Compose file, Runtime creates an Aiven for PostgreSQL service for you, and you can set its plan and region as part of the deployment. If you already have a PostgreSQL service, you can select that instead.

The DAG files live in the dags/ directory in the repository, which means they're under version control, and you can branch, review and revert them like any other code.

Deploy it

1. Make your own copy of the template repository.
2. Add a DAG file to the dags/ directory. The Astronaut ETL example DAG is a good first choice: it's engaging and has no extra dependencies.
3. Commit and push to your repository.
4. Follow the Runtime deployment documentation. Connect your GitHub account to your Aiven organization if this is your first deployment, then select your repository and branch.
5. Select compose.aiven.yaml and scan it.
6. On the Airflow card, set USERNAME and PASSWORD, and choose a plan with at least 2 vCPUs and 4 GB RAM.
7. Click Deploy.

When the application is running, click the Application URL and log in with the username and password you set.

Be patient on first start. Airflow takes several minutes to come up, and the container can show "Running" before the web server answers. A 502 Bad Gateway in that window is not unusual while Airflow is still starting. Wait a few minutes, then check the runtime logs and increase the RAM if the error persists.

The official Airflow Docker guide recommends 4 GB as a minimum and 8 GB for smoother operation. Standalone mode runs the web server, scheduler, triggerer, DAG processor and API server in one container, and that takes memory.

What to use it for

This is a good way to try Airflow in a real cloud environment, instead of running a local Docker container.

The core Airflow features all work, including scheduled jobs, multi-step pipelines, and workflows that wait on conditions or external data.

In an Aiven context, other possibilities include:

• Tasks that call the Aiven REST API to create, power off, power on and delete Runtime applications. A nightly DAG can switch off a development environment and bring it back each morning.
• Tasks that coordinate Runtime applications with each other. For instance, an application that sets a value in Valkey which a DAG then reacts to, or the other way round.

Limitations

The template makes three deliberate trade-offs.

• It runs Standalone Airflow. It runs everything in one container, all tasks on a single machine, with no distributed workers. This suits development and moderate workloads. The Airflow documentation recommends running production-scale deployments on Kubernetes, and the template doesn't try to address that. If you are interested in Airflow on Kubernetes, look at the Airflow Helm chart.

• It uses the Simple Auth Manager. You set the USERNAME and PASSWORD for your admin user as environment variables, which the template writes to a JSON file inside the running container. Airflow then reads the credentials from that file. If you don't set both values, Airflow generates an admin user with a random password and writes both the username and password to the runtime log.

  The Simple auth manager is intended for development and testing. For anything more, read the Auth manager overview, which explains how auth managers plug in and links to the auth managers provided by Airflow providers, and the Airflow security documentation.

• The logs are ephemeral. For persistent logs, configure external logging, for instance using Aiven for OpenSearch.

Try it for yourself

• Fork the aiven-runtime-airflow template.
• Deploy the Astronaut ETL DAG and watch it run in the Airflow UI.
• Write a DAG that calls the Aiven API to stop and start one of your Runtime applications on a schedule.
• Swap the Airflow image version in the Dockerfile, or add providers such as PostgreSQL and HTTP - see Adding new PyPI packages individually and Adding packages from requirements.txt.

If you want to see how another workload deploys on Runtime, read Deploying Apache Kafka® Streams next to your data.

For more on other Runtime templates that can be used together, see Build your open and agnostic agent stack.