Replicated SQLite using the Raft consensus protocol.

The world's most popular open source database.

A distributed NewSQL database compatible with MySQL protocol.

A free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.

Pinterest MySQL Management Tools.

An enhanced, drop-in replacement for MySQL.

The world's most advanced open source database.

Managed PostgreSQL with pgvector for AI workloads. HNSW indexing, sub-4ms latency, and built-in SQL editor with…

Makes it easy to set up, operate, and scale a relational database in the cloud.

Scalable SQL database with the NOSQL goodies.

An open source, BSD licensed, advanced key-value cache and store.

A distributed database designed to deliver maximum data availability by distributing data across multiple servers.

A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency…

A scalable, searchable key-value store. Deprecated.

A high performance NoSQL database supporting many data structures, an alternative to Redis.

A lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency.

A key-value store for microcontroller and IoT applications.

The right choice when you need scalability and high availability without compromising performance.

This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is…

A script to easily create and destroy an Apache Cassandra cluster on localhost.

NoSQL data store using the seastar framework, compatible with Apache Cassandra.

The Hadoop database, a distributed, scalable, big data store.

A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data…

Distributed. Columnar. Versioned. Streaming. SQL.

Distributed, MPP columnar database with extensive analytics SQL.

Distributed columnar DBMS for OLAP. SQL.

An open-source, document database designed for ease of development and scaling.

Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB®…

Distributed Transactional In-Memory Database (based on MongoDB).

Search & Analyze Data in Real Time.

The highest performing NoSQL distributed database.

The open-source database for the realtime web.

Fully Transactional NoSQL Document Database.

Open-source multi-model database with native graph, document, key-value, and vector support. SQL, Cypher, and Gremlin…

The world's leading graph database.

Typed graph database where agents branch and merge like Git. S3-native, Rust, traversal + vector + BM25 in one runtime.

2nd Generation Distributed Graph Database with the flexibility of Documents in one product with an Open Source…

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values.

A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and…

A distributed, fault-tolerant graph database by Twitter. Deprecated.

A database for user interactions (likes, views, follows) represented as graphs, with precomputed reads served in…

The fully transactional, cloud-ready, distributed database.

An open source, distributed, in-memory database for scale-out applications.

A large-scale graph database.

Scalable datastore for metrics, events, and real-time analytics.

A scalable, distributed Time Series Database.

A relational column-oriented database designed for real-time analytics on time series and event data.

Fast scalable time series database.

A scalable time series database based on Cassandra and Elasticsearch, by Spotify.

Column oriented distributed data store ideal for powering interactive applications.

Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data.

A numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word…

A time-series object store for Cassandra that handles all the complexity of building wide row indexes.

Fast distributed metrics database.

A distributed system designed to ingest and process time series data.

A time series database application that provides secure access to time series data based on Accumulo and Grafana.

An in-memory database and application server.

The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and…

An open-source graph database. Google.

OLTP + OLAP Database built on Apache Spark.

Built as an extension on top of PostgreSQL, TimescaleDB is a time-series SQL database providing fast analytics,…

A fast in-process analytical database that has zero external dependencies, runs on Linux/macOS/Windows, offers a rich…

In-process analytical SQL database written in C++20. Reads Parquet, CSV, JSON, Avro, Arrow, SQLite, and Excel…

Embedded ClickHouse — full ClickHouse SQL dialect, ~80 data formats, and 12+ source connectors (S3, Postgres, MongoDB,…

An embedded vector database for on-device RAG and edge AI, the SQLite of vector databases.

High-performance blob and time-series storage, with edge deployment, selective replication, and efficient querying of…

An open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.

A Python library that facilitates the comparison of two DataFrames in Pandas, Polars, Spark and more. The library goes…

Data Validation Tool compares data from source and target tables to ensure that they match. It provides column…

A high-performance Python library for comparing large datasets (CSV, Parquet) locally using Rust and Polars. It…

Python SDK that dispatches parallel web-research agents across table rows, synthesizing multi-agent findings into…

Data ingestion engine that connects 400+ Singer taps to Parquet files in cloud buckets (S3, GCS, Azure). Streaming,…

Managed event ingestion service that converts JSON sent to a REST API into Hive-partitioned Parquet on Cloudflare R2,…

CLI tool to enrich CSV files with company data (financials, contacts, metadata) from 250M+ company records. Available…

CLI tool to copy data between databases with a single command. Supports 50+ sources including PostgreSQL, MySQL,…

Publish-subscribe messaging rethought as a distributed commit log.

Change data capture from PostgreSQL into Kafka. Deprecated.

Simplified command-line administration for Kafka brokers.

Generic command line non-JVM Apache Kafka producer and consumer.

A PostgreSQL extension to produce messages to Apache Kafka.

The Apache Kafka C/C++ library.

Kafka in Docker.

A tool for managing Apache Kafka.

Node.js client for Apache Kafka 0.8.

Pinterest's Kafka to S3 distributed consumer.

Kafka-winston logger for Node.js from Uber.

A Kafka Proxy, solving problems like encrypting your Kafka data at rest.

A fully managed, cloud-based service for real-time data processing over large, distributed data streams.

Robust messaging for applications.

A fast&simple pipeline building library for Python data devs, runs in notebooks, cloud functions, airflow, etc.

OSS Reverse ETL CLI. Sync data from warehouses to business tools via YAML.

An open source data collector for unified logging layer.

An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud…

A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as…

Data Acquisition and Processing Made Easy. Deprecated.

Universal data ingestion framework for Hadoop from LinkedIn.

An open source event messaging platform that provides a REST API on top of Kafka-like queues.

Provides a new storage abstraction - a stream - for continuous and unbounded data.

An open-source distributed pub-sub messaging system.

Utility belt to handle data on AWS.

Open-source data integration for modern data teams.

Self-hosted database migration and change data capture (CDC) tool with built-in SQL IDE.

Real-time data ingestion tool leveraging change data capture.

CLI data integration tool specialized in moving data between databases, as well as storage systems.

CLI & code-first ELT.

The fastest way to build custom data extractors and loaders compliant with the Singer Spec.

Live import all your Google Sheets to your data warehouse.

A delimited data preboarding framework that fills the gap between MFT and the data lake.

No/low-code data pipeline platform that handles both batch and real-time data ingestion.

Lightweight Node.js ETL framework for databases → data lakes/warehouses.

High-performance, streaming-first ETL engine for Node.js and TypeScript with constant memory footprint.

Polyglot document intelligence library with a Rust core and bindings for Python, TypeScript, Go, and more. Extracts…

Python PDF-to-Markdown orchestrator. Classifies each page and routes to the optimal backend (PyMuPDF, Docling,…

Managed cloud object storage transfers for ingestion workflows.

Real-time X (Twitter) data extraction platform with REST API (76 endpoints), 20 bulk extraction tools, account…

High-speed CLI tools for database export, import, replication and migration with parallel streaming to CSV, Parquet,…

A real-time B2B data API for company and people intelligence, providing firmographics, headcount signals, job…

Conflict-free merge for DataFrames, JSON, ML models & distributed agents — powered by CRDTs.

Crawlee-based actor extracting structured LinkedIn job listings at scale without API keys.

Context-Aware RAG Processing Queue for high availability and adaptive rate-limiting.

Local-first, open-source desktop ETL/ELT studio: drag a pipeline onto a canvas (or describe it to a built-in on-device…

Open-source self-hosted game analytics pipeline. HTTP event collector with NATS JetStream buffering, raw Parquet in…

Config-driven data-movement platform for Rust with pluggable source and sink connectors, running ETL, CDC, and…

An open-source Customer Data Platform. Captures event data from websites, apps, and servers and streams it into…

Open-source, Java-based ETL and script execution tool for transferring and transforming data between databases, files,…

A distributed file system designed to run on commodity hardware.

A pure python HDFS client.

Object storage built to retrieve any amount of data from anywhere.

Utils for streaming large files (S3, HDFS, gzip, bz2).

A memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks,…

A unified, distributed storage system designed for excellent performance, reliability, and scalability.

A high-performance Cloud-Native file system driven by object storage for large-scale data storage.

Orange File System is a branch of the Parallel Virtual File System.

A bite-sized, lightweight HDFS compatible file system built over Cassandra.

Gluster Filesystem.

Fault-tolerant distributed file system for all storage needs.

Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of…

A file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.

Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file…

The AI native file format. Trust scores, source provenance, and compliance metadata that embed into 20+ formats (DOCX,…

Apache Avro™ is a data serialization system.

A columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data…

A fast compressor/decompressor. Used with Parquet.

A parallel implementation of gzip for modern multi-processor, multi-core machines.

The smallest, fastest columnar storage for Hadoop workloads.

The Apache Thrift software framework, for scalable cross-language services development.

Protocol Buffers - Google's data interchange format.

A flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats.

A fast and efficient object graph serialization framework for Java.

Specialized JSONL log compressor with block-level timestamp indexing and DuckDB integration. Achieves ~9% compression…

Browser-based viewer, SQL workbench and converter for Parquet files powered by DuckDB-WASM. Fully client-side, no…

A unified programming model that implements both batch and streaming data processing jobs that run on many execution…

Makes it easy to build scalable fault-tolerant streaming applications.

A streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed…

A free and open source distributed realtime computation system.

A distributed stream processing framework.

An easy to use, powerful, and reliable system to process and distribute data.

An open source framework for managing storage for real time processing, one of the most interesting feature is the…

An open source ETL framework to build fresh index for AI.

An ACID-compliant RDBMS which uses a shared nothing architecture.

The Streaming SQL Database.

Streaming and tasks execution between Spring Boot apps.

A data-processing toolkit for python 3.5+.

Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing.

The streaming database built for IoT data storage and real-time processing.

An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of…

An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE,…

A framework for building real-time streaming data processing applications that supports a wide range of ingestion…

Performant open-source Python ETL framework with Rust runtime, supporting 300+ data sources.

A software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) -…

A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or…

A community index of packages for Apache Spark.

Connecting Apache Spark with different data stores. Deprecated.

Examples by Zhen He.

The REST Spark Server.

A free & cross platform monitoring tool (Spark UI / Spark History Server alternative).

A web service that makes it easy to quickly and cost-effectively process vast amounts of data.

A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective.

An application framework which allows for a complex directed-acyclic-graph of tasks for processing data.

A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a…

A cloud native data pipeline and transformation toolkit written in Go.

Personal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks,…

Fast scalable machine learning API for smarter applications.

An environment for quickly creating scalable performant machine learning applications.

Spark's scalable machine learning library consisting of common learning algorithms and utilities, including…

Pure-Go classic machine learning toolkit and data engineering utilities. Eight algorithms with zero external…

Open source Master Data Management platform using machine learning for entity resolution at scale. Native to…

A machine learning platform that enables data scientists and app developers to easily create intelligent apps at scale.

An iterative graph processing system built for high scalability.

Apache Spark's API for graphs and graph-parallel computation.

A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data…

Data warehouse software facilitates querying and managing large datasets residing in distributed storage.

Scalable machine learning library for Hive/Hadoop.

Python interface to Hive and Presto.

Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.

A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or…

Fast JavaScript charts for any data set.

D3-based reusable chart library.

A JavaScript library for manipulating documents based on data.

D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in.

A JavaScript Charting Library for Streaming Data.

Python helpers for building dashboards using Flask and React.

Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python.

A modern, enterprise-ready business intelligence web application.

Make Your Company Data Driven. Connect to any data source, easily visualize and share your data.

The easy, open source way for everyone in your company to ask questions and learn from data.

Open-source, self-hosted, warehouse-native product analytics. Runs funnels, retention, and paths directly on DuckDB,…

A pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics /…

A Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive…

Natural language database query interface with automatic chart generation, supporting Chinese and English queries.

Agentic AI platform to connect any database (PostgreSQL, MySQL, MongoDB, etc.) and query in plain English; includes…

Open-source SQL to map platform for BigQuery, Snowflake, and PostGIS.

Open-source analytics notebook for reusable SQL workflows, interactive reports, and AI-assisted data exploration.

Open-source Python library for interactive cross-filter dashboards on large datasets. Zoom, pan, and selection are…

Governed, multi-tenant MCP access to your customers' data. Turn your warehouse, dbt, or semantic layer into a secure,…

Intent-as-code workflow engine for AI data pipelines: reviewable YAML DAGs statically checked (schema, permits, cost…

Open-source semantic sidecar that compiles YAML-defined dimensions, measures, and metrics into optimized SQL across 8…

End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single…

Open-source platform for data preparation, synthetic data generation, and AI/data pipelines. Includes reusable skills…

A Python module that helps you build complex pipelines of batch jobs.

An application cron-like system. Used w/Luigi. Deprecated.

Java based application development platform.

A system to programmatically author, schedule, and monitor data pipelines.

A batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job…

A workflow scheduler system to manage Apache Hadoop jobs.

DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs.

An open-source Python library for building data applications.

A lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL…

A framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates,…

An open-source framework and web based IDE to manage datasets and their dependencies. SQLX extends your existing SQL…

A lightweight Python library for building execution pipelines with retry, parallel execution, cron scheduling, and…

A reverse-ETL tool that let you sync data from your cloud data warehouse to SaaS applications like Salesforce,…

A command line tool that enables data analysts and engineers to transform data in their warehouses more effectively.

Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows…

A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS…

An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed,…

Self-hosted gateway for safe, auditable queries for agents across approved data sources.

An orchestration and observability platform. With it, developers can rapidly build and scale resilient code, and…

The open-source reverse ETL, data activation platform for modern data teams.

Create automated workflows and logic using API's for your notification service. Add templates, batching, preferences,…

Open-source data pipeline tool for transforming and integrating data.

An open-source data transformation framework for managing, testing, and deploying SQL and Python-based data pipelines…

An open source platform that delivers resilience and manageability to object-storage based data lakes.

A Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables.

A modular Data Lakehouse platform that simplifies the management and monitoring of Apache Spark clusters across…

An open-source, unified metadata management for data lakes, data warehouses, and external catalogs.

FlightPath is a gateway to a data lake's bronze layer, protecting it from invalid external data file feeds as a…

Managed lakehouse platform on Apache Iceberg with DuckDB query compute, S3 storage, Postgres wire protocol, and SQL…

A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2).

JDBC importer for Elasticsearch.

PostgreSQL Extension that allows creating an index backed by Elasticsearch.

Package golang service into minimal Docker containers.

Easily manage Docker containers & their data.

RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers.

Application Containers for Masses.

Weaving Docker containers into applications.

A lightweight tool for easy deployment and rollback of dockerized applications.

Analyzes resource usage and performance characteristics of running containers.

Docker microservice for saving/restoring volume data to S3.

Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated.

A cluster manager, designed for both long-lived services and short-lived batch processing workloads.

Visualize Docker images and the layers that compose them.

DEX data via SSE streaming across 36 blockchains. 36M+ pools, 33M+ tokens. Metered free tier, no API key required,…

Remote MCP server for real-time financial data, 3.2M+ news articles, ML options pricing, and news bias analysis. Free,…

The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data.

Real-time X (Twitter) data API providing tweets, profiles, search, communities and engagement metrics. Up to 50x…

Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.

Data generation platform for producing synthetic event streams with complex correlations.

Real-time data is available including comments, submissions and links posted to reddit.

GitHub's public timeline since 2011, updated every hour.

Open source repository of web crawl data.

Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw…

A 30-metro composite of US household cost burdens (housing, taxes, childcare, healthcare, transport) aggregated from…

The world's most comprehensive authoritative data source knowledge base. 160+ curated sources from governments,…

42-table synthetic SME dataset with double-entry accounting, tax compliance (AU/US/UK), and temporal realism. CSV,…

Synthetic financial savings behavior generator for Latin America: users, savings goals, and transactions calibrated on…

An open-source service monitoring system and time series database.

Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption.

Intent signal monitoring CLI. Track LinkedIn engagers, keyword posters, job changers, funding events. JSON output for…

The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy.

A general-purpose open-source data profiler for high-level analysis of a dataset.

An open-source data profiler specifically focused on discovery and validation of complex patterns in data.

Open-source and free relational database schema discovery and comprehension tool. Documents and diagrams relational…

Open-source agentic data quality framework with LLM-powered diagnosis, root-cause analysis, SQL auto-fix proposals,…

A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These…

An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying…

Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and…

Open Source data validation framework to manage data quality. Users can define and document “expectations” rules about…

A vendor-neutral, declarative data quality engine. Define checks in YAML, run anywhere. Includes 16 built-in check…

Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema…

Free online SQL playground for MySQL, PostgreSQL, and SQL Server. Create database structures, run queries, and share…

Write, run, and test PySpark code on Spark Playground's online compiler. Access real-world sample datasets & solve…

Decorator-first DataFrame contracts/validation (columns/dtypes/constraints) at function boundaries. Supports…

A Snowflake-compatible emulator for local development and testing.

Real-time data quality firewall for pipelines and APIs. Screens rows in milliseconds for schema drift, null spikes,…

Interview practice with SQL query execution, Python, and data modeling exercises.

JSON/XML validation and API contract monitoring tool for debugging and testing structured data.

Open-source tool that scans SQL databases for sensitive columns, masks them with deterministic fakes, and validates…

News, tips, and background on Data Engineering.

Subreddit focused on ETL.

Job board focused on AI, ML, and data engineering roles with 7,400+ listings, salary data, and a free REST API.

The first technical conference that bridges the gap between data scientists, data engineers and data analysts.

Interviews with AI and data infrastructure leaders on building production systems.

The show about modern data infrastructure.

Technical deep dives on AI engineering, from model training to deployment.

Making AI practical, productive, and accessible to everyone.

Daily interviews about technical software topics, including data infrastructure.

How analytics engineers build and maintain data pipelines at scale.

A show where they talk to data engineers, analysts, and data scientists about their experience around building and…

A practical introduction to data engineering on the Snowflake cloud data platform.

This blog offers a curated list of top data science books, categorized by topics and learning stages, to aid readers…

A guide to designing an Apache Iceberg lakehouse from scratch.

A fast, friendly guide to integrating large language models into your data workflows.