Awesome Apache Airflow
Section: Vital links · (also the official website)
FreshScore 82
Entry
Apache Airflow
Appears in 9 awesome lists
is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…
Found in these lists
Awesome High Performance Computing
Section: Software · A platform to programmatically author, schedule, and monitor workflows.
FreshScore 78
Awesome LLMOps
Section: Workflow · A platform to programmatically author, schedule and monitor workflows.
ActiveScore 75
Awesome MLOps
Section: Data Processing · Platform to programmatically author, schedule, and monitor workflows.
FreshScore 80
Awesome Scalability
Section: Scalability
SlowScore 69
Awesome-Selfhosted
Section: Automation · Platform to programmatically author, schedule, and monitor workflows. (Source Code) Apache-2.0 Python/Docker
FreshScore 94
Awesome Software Engineering for Machine Learning
Section: Tooling · Programmatically author, schedule and monitor workflows.
StaleScore 51
Table of Contents
Section: ML frameworks & applications · is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…
SlowScore 61
Table of Contents
Section: ML Frameworks, Libraries, and Tools · is an open-source workflow management platform created by the community to programmatically author, schedule and monitor workflows. Install. Principles. Scalable. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to…
SlowScore 51
Often listed next to it
Docker
is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly working in collaboration with cloud, Linux, and Windows vendors, including Microsoft.
In 18 listsDetails
Apache Airflow
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
In 13 listsDetails
Prometheus
is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.
In 13 listsDetails
Prefect
Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.
In 11 listsDetails
Hamilton
| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.
In 10 listsDetails
Argo Workflows
Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).
In 10 listsDetails
Vaex
Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).
In 9 listsDetails
Kubernetes
is an open-source system for automating deployment, scaling, and management of containerized applications.
In 10 listsDetails