Awesome Big Data
Section: Data Ingestion · a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
ActiveScore 84
Entry
Apache Pulsar
Appears in 6 awesome lists
a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
Open github.comapache/pulsar
Found in these lists
Awesome Cloud Native
Section: Streaming & Messaging · A distributed messaging and streaming platform.
FreshScore 87
Awesome Data Analysis
Section: Tools · A cloud-native, distributed messaging and streaming platform.
FreshScore 80
Awesome Integration
Section: Message Broker · Versatile pub/sub and streaming platform offering scalable, low-latency messaging for modern applications.
FreshScore 82
Awesome Java
Section: Messaging · Apache-2.0 🟢Distributed pub/sub-messaging system.
FreshScore 93
Awesome Ops
Section: 消息队列 · Pulsar 是一个分布式 pub-sub 消息传递平台，具有非常灵活的消息传递模型和直观的客户端 API。
ActiveScore 79
Often listed next to it
Luigi
Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.
In 15 listsDetails
Apache Airflow
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
In 13 listsDetails
Dagster
Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.
In 12 listsDetails
Prefect
Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.
In 11 listsDetails
Apache Spark
Unified analytics engine for large-scale data processing. In-memory cluster computing with high-level APIs in Python, Scala, Java, and R. Powers MLlib for distributed machine learning and Structured Streaming for real-time data. Apache 2.0 licensed.
In 8 listsDetails
NATS
star:19047 NATS is a simple, secure, and performant communications system for digital systems, services, and devices.
In 7 listsDetails
Kestra
Event-driven orchestration and scheduling platform for mission-critical workflows. Infrastructure-as-Code approach with declarative YAML, Git version control integration, and hundreds of plugins for data pipelines and ML workflows. Apache 2.0 licensed.
In 8 listsDetails
Apache Flink
Stream processing framework with powerful batch and streaming capabilities. High-throughput, low-latency runtime with exactly-once processing guarantees. Ideal for real-time AI inference pipelines and event-driven ML applications. Apache 2.0 licensed.
In 7 listsDetails