Awesome Data Engineering
Section: Workflow · The open-source reverse ETL, data activation platform for modern data teams.
Entry
Appears in 4 awesome lists
(label: good first issue) The open-source reverse ETL, data activation platform for modern data teams.
Section: Workflow · The open-source reverse ETL, data activation platform for modern data teams.
Section: Ruby · (label: good first issue) The open-source reverse ETL, data activation platform for modern data teams.
Section: Open Source Rails Apps · The open-source reverse ETL, data activation platform for modern data teams. (using Rails 7.1.1) - 🌍
Section: Data Processing and ETL · The open-source reverse ETL, data activation platform developed using Ruby and Ruby on Rails.
Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.
| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.
Event-driven orchestration and scheduling platform for mission-critical workflows. Infrastructure-as-Code approach with declarative YAML, Git version control integration, and hundreds of plugins for data pipelines and ML workflows. Apache 2.0 licensed.
(label: good first issue) Ruby on Rails (Rails) is an open source web application framework written in Ruby.