Apache Airflow

Apache Airflow is an open-source platform for authoring, scheduling and monitoring workflows as code. Pipelines are defined in Python, including dynamic task generation, with Jinja templating for parametrization. A web UI shows the status and logs of running and completed tasks. Its modular architecture uses a message queue to coordinate any number of workers, and users can write custom operators. Prebuilt operators connect to Google Cloud, AWS, Microsoft Azure and other services. It suits teams with Python skills building data pipelines and ML workflows.