Apache Hadoop

Apache Hadoop

Apache Hadoop Ozone

An Object Store for Apache Hadoop

Apache Tez

A Framework for YARN-based, Data Processing Applications In Hadoop

SpatialHadoop

SpatialHadoop is a MapReduce extension to Apache Hadoop designed specially to work with spatial data.

GIS Tools for Hadoop

Big Data Spatial Analytics for the Hadoop Framework

Elasticsearch Hadoop

Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive and Apache Pig.

hadoopy

Python MapReduce library written in Cython.

mrjob

mrjob is a Python 2.5+ package that helps you write and run Hadoop Streaming jobs.

pydoop

Pydoop is a package that provides a Python API for Hadoop.

hdfs-du

HDFS-DU is an interactive visualization of the Hadoop distributed file system.

White Elephant

Hadoop log aggregator and dashboard

Genie

Genie provides REST-ful APIs to run Hadoop, Hive and Pig jobs, and to manage multiple Hadoop resources and perform job submissions across them.

Apache Kylin

Apache Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets

Crunch

Go-based toolkit for ETL and feature extraction on Hadoop

Apache Ignite

Distributed in-memory platform

Apache Slider

Apache Slider is a project in incubation at the Apache Software Foundation with the goal of making it possible and easy to deploy existing applications onto a YARN cluster.

Apache Twill

Apache Twill is an abstraction over Apache Hadoop® YARN that reduces the complexity of developing distributed applications, allowing developers to focus more on their application logic.

mpich2-yarn

Running MPICH2 on Yarn

Apache HBase

Apache HBase

Apache Phoenix

A SQL skin over HBase supporting secondary indices

happybase

A developer-friendly Python library to interact with Apache HBase.

Hannibal

Hannibal is tool to help monitor and maintain HBase-Clusters that are configured for manual splitting.

Haeinsa

Haeinsa is linearly scalable multi-row, multi-table transaction library for HBase

hindex

Secondary Index for HBase

Apache Accumulo

The Apache Accumulo™ sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.

OpenTSDB

The Scalable Time Series Database

Apache Cassandra

Column-oriented and providing high availability with no single point of failure.

Apache Hive

The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL

Apache Phoenix

A SQL skin over HBase supporting secondary indices

Apache HAWQ (incubating)

Apache HAWQ is a Hadoop native SQL query engine that combines the key technological advantages of MPP database with the scalability and convenience of Hadoop

Lingual

SQL interface for Cascading (MR/Tez job generator)

Apache Impala

Apache Impala is an open source massively parallel processing (MPP) SQL query engine for data stored in a computer cluster running Apache Hadoop. Impala has been described as the open-source equivalent of Google F1, which inspired its development in 2012.

Presto

Distributed SQL Query Engine for Big Data. Open sourced by Facebook.

Apache Tajo

Data warehouse system for Apache Hadoop

Apache Drill

Schema-free SQL Query Engine

Apache Trafodion

Transactional SQL-on-Hadoop/HBase.

Apache Calcite

A Dynamic Data Management Framework

Apache Atlas

Metadata tagging & lineage capture suppoting complex business data taxonomies

Apache Kudu

Kudu provides a combination of fast inserts/updates and efficient columnar scans to enable multiple real-time analytic workloads across a single storage layer, complementing HDFS and Apache HBase.

Confluent Schema registry for Kafka

Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas.

Hortonworks Schema Registry

Schema Registry is a framework to build metadata repositories.

Apache Oozie

Apache Oozie

Azkaban

Apache Falcon

Data management and processing platform

Apache NiFi

A dataflow system

Apache AirFlow

Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines

Luigi

Python package that helps you build complex pipelines of batch jobs

Apache Flume

Apache Flume

Suro

Netflix's distributed Data Pipeline

Apache Sqoop

Apache Sqoop

Apache Kafka

Apache Kafka

Gobblin from LinkedIn

Universal data ingestion framework for Hadoop

Apache Pig

Apache Pig

Apache DataFu

A collection of libraries for working with large-scale data in Hadoop

vahara

Machine learning and natural language processing with Apache Pig

packetpig

Open Source Big Data Security Analytics

akela

Mozilla's utility library for Hadoop, HBase, Pig, etc.

seqpig

Simple and scalable scripting for large sequencing data set(ex: bioinfomation) in Hadoop

Lipstick

Pig workflow visualization tool. Introducing Lipstick on A(pache) Pig

PigPen

PigPen is map-reduce for Clojure, or distributed Clojure. It compiles to Apache Pig, but you don't need to know much about Pig to use it.

Kite Software Development Kit

A set of libraries, tools, examples, and documentation

gohadoop

Native go clients for Apache Hadoop YARN.

Hue

A Web interface for analyzing data with Apache Hadoop.

Apache Zeppelin

A web-based notebook that enables interactive data analytics

Apache Thrift

The Apache Thrift software framework, for scalable cross-language services development.

Apache Avro

Apache Avro is a data serialization system.

Elephant Bird

Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.

Spring for Apache Hadoop

hdfs - A native go client for HDFS

Oozie Eclipse Plugin

A graphical editor for editing Apache Oozie workflows inside Eclipse.

snakebite

A pure python HDFS client

Apache Parquet

Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Apache Superset (incubating)

Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

Schema Registry UI

Web tool for the Confluent Schema Registry in order to create / view / search / evolve / view history & configure Avro schemas of your Kafka cluster.

Apache Storm

framework for stream processing by Twitter also on YARN.

Apache Samza

stream processing framework, based on Kafka and YARN.

Apache Spark

Apache Flink

Apache Flink is a platform for efficient, distributed, general-purpose data processing. It supports exactly once stream processing.

Apache Pulsar (incubating)

Apache Pulsar (incubating) is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication.

Apache Druid (incubating)

A high-performance, column-oriented, distributed data store.

Apache Spark

Apache Spark™ is a fast and general engine for large-scale data processing. Subproject: MLlib. License: Apache 2.

Spark Packages

A community index of packages for Apache Spark

SparkHub

A community site for Apache Spark

Apache Crunch

a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce.

Cascading

Cascading is the proven application development platform for building data applications on Hadoop.

Apache Flink

Apache Flink is a platform for efficient, distributed, general-purpose data processing.

Apache Apex (incubating)

Enterprise-grade unified stream and batch processing engine.

Apache Livy (incubating)

Apache Livy (incubating) is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. With Livy, new applications can be built on top of Apache Spark that require fine grained interaction with many Spark contexts.

Apache Bigtop

Apache Bigtop: Packaging and tests of the Apache Hadoop ecosystem

Apache Ambari

Apache Ambari

Ganglia Monitoring System

ankush

A big data cluster management tool that creates and manages clusters of different technologies.

Apache Zookeeper

Apache Zookeeper

Apache Curator

ZooKeeper client wrapper and rich ZooKeeper framework

inviso

Inviso is a lightweight tool that provides the ability to search for Hadoop jobs, visualize the performance, and view cluster utilization.

Logit.io

Send logs from Hadoop to Elasticsearch for monitoring and alerting.

ElasticSearch

is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.

Apache Solr

Apache Solr is an open source search platform built upon a Java library called Lucene.

Banana

Kibana port for Apache Solr

Apache Nutch

Apache Nutch is a highly extensible and scalable open source web crawler software project.

Apache Ranger

Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform.

Apache Sentry

An authorization module for Hadoop

Apache Knox Gateway

A REST API Gateway for interacting with Hadoop clusters.

Big Data Benchmark

Benchmark of Redshift, Hive, Shark, Impala and Stiger/Tez.

HiBench

a Hadoop benchmark suite.

YCSB

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of NoSQL database management systems.

Apache Mahout

An Apache-backed machine learning library for Hadoop.

Oryx 2

Lambda architecture on Spark, Kafka for real-time large scale machine learning

MLlib

MLlib is Apache Spark's scalable machine learning library.

R

R is a free software environment for statistical computing and graphics.

RHadoop

including RHDFS, RHBase, RMR2, plyrmr

Apache Lens

Apache SINGA (incubating)

SINGA is a general distributed deep learning platform for training big deep learning models over large datasets

BigDL

BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

Apache Hivemall (incubating)

Apache Hivemall is a scalable machine learning library that runs on Apache Hive, Spark and Pig.

https://github.com/edwardcapriolo/hive_cassandra_udfs

https://github.com/livingsocial/HiveSwarm

https://github.com/ThinkBigAnalytics/Hive-Extensions-from-Think-Big-Analytics

Elephant Bird

Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.

https://github.com/lovelysystems/ls-hive

https://github.com/klout/brickhouse

https://github.com/dvasilen/Hive-Cassandra

https://github.com/yc-huang/Hive-mongo

https://github.com/balshor/gdata-storagehandler

https://github.com/chimpler/hive-solr

https://github.com/bfemiano/accumulo-hive-storage-manager

https://github.com/forward3d/rbhive

https://github.com/synctree/activerecord-hive-adapter

https://github.com/hrp/sequel-hive-adapter

https://github.com/forward/node-hive

https://github.com/recruitcojp/WebHive

shib

WebUI for query engines: Hive and Presto

https://github.com/dmorel/Thrift-API-HiveClient2

(Perl - HiveServer2)

PyHive

Python interface to Hive and Presto

https://github.com/recruitcojp/OdbcHive

HiveRunner

An Open Source unit test framework for hadoop hive queries based on JUnit4

Beetest

A super simple utility for testing Apache Hive scripts locally for non-Java developers.

Hive_test

Unit test framework for hive and hive-service

Flume MongoDB Sink

Flume RabbitMQ source and sink

Flume UDP Source

.Net FlumeNG Clients

Hadoop Weekly

The Hadoop Ecosystem Table

Hadoop illuminated

Open Source Hadoop Book

AWS BigData Blog

Hadoop360

How to monitor Hadoop metrics

Guide to monitoring Hadoop, with an overview of Hadoop architecture, and native methods for metrics collection.

Apache Hadoop In Theory And Practice

Hadoop Operations at LinkedIn

Hadoop Performance at LinkedIn

Docker based Hadoop provisioning

Hadoop: The Definitive Guide

Hadoop Operations

Apache Hadoop Yarn

HBase: The Definitive Guide

Comprehensive guide to HBase.

Programming Pig

Programming Hive

Hadoop in Practice, Second Edition

Hadoop in Action, Second Edition

ApacheCon

Strata + Hadoop World

DataWorks Summit

Spark Summit