Awesome Data Engineering
Section: Data Ingestion · An open source data collector for unified logging layer.
Entry
Appears in 5 awesome lists
Set Up Once, Collect More. Fluentd is an open source data collector designed for processing data streams. 150+ plugins instantly enable you to store the data for Log Management, Big Data Analytics, etc
Section: Data Ingestion · An open source data collector for unified logging layer.
Section: Log Management · Data collector for unified logging layer. (Source Code) Apache-2.0 Ruby
Section: Log Management · Log Collector and Shipper.
Section: Log Monitoring · Set Up Once, Collect More. Fluentd is an open source data collector designed for processing data streams. 150+ plugins instantly enable you to store the data for Log Management, Big Data Analytics, etc
Section: Observability & Monitoring · Data collector for unified logging layer.
CLI tool to copy data from any source to any destination with a single command. Load data into your warehouse before dbt transforms it. Supports 50+ sources including Postgres, MongoDB, Salesforce, Shopify.
Real-time X (Twitter) data platform for tweet search, user lookup, follower/following extraction, engagement metrics, account monitoring, reply/retweet/quote extraction, community & Space data, and mutual follow checks.
Local-first, open-source desktop ETL/ELT studio: drag a pipeline onto a canvas (or describe it to a built-in on-device AI assistant) and run it at native speed through DuckDB. 290+ connectors, a scheduler, and an MCP server for driving pipelines from an LLM. No cloud, no servers.
Distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. (Source Code) Apache-2.0 Java
An open-source Python package that extends the power of Pandas library to AWS connecting DataFrames and AWS data related services (Amazon Redshift, AWS Glue, Amazon Athena, Amazon EMR, etc).
Publish-subscribe messaging rethought as a distributed commit log.
managed cloud object storage transfers for data ingestion workflows.
Open source Erlang-based message broker that just works.