Awesome Data Engineering
Section: Stream Processing · An open source ETL framework to build fresh index for AI.
FreshScore 87
Entry
GitHub
Appears in 6 awesome lists
ETL framework to build fresh context for AI agents, with incremental processing
Open github.comcocoindex-io/cocoindex
Found in these lists
Awesome Pipeline
Section: Pipeline frameworks & libraries · ETL framework to build fresh index.
FreshScore 86
awesome-python
Section: LLM and Inference · Incremental engine for long horizon agents Star if you like it!
FreshScore 81
Awesome Vector Search
Section: Library
ActiveScore 73
Best Of Python
Section: CocoIndex (24 · 6.1K) - Data transformation framework for AI. Ultra performant, with.. Apache-2 · (👨💻 93 · 🔀 450 · 📦 73):
FreshScore 88
Awesome Rust
Section: MLOps · ETL framework to build fresh context for AI agents, with incremental processing
FreshScore 94
Often listed next to it
LangChain
Langchain integrates various providers like Anthropic, AWS, and OpenAI, and offers tools for components such as LLMs, chat models, and data analysis, supporting functionalities from Alpha Vantage to YouTube github | docs
In 20 listsDetails
LiteLLM
Unified proxy and SDK that routes to 100+ LLM providers behind a single OpenAI-compatible interface, with a Router handling retry/fallback across deployments, per-project cost and rate-limit tracking, and OTEL callback integrations. The right infrastructure layer when your harness needs provider…
In 16 listsDetails
Luigi
Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.
In 15 listsDetails
Dify
February 2026 release making human oversight a native workflow primitive: suspend execution at critical decision points, expose review-and-edit UI mid-flow, and route subsequent execution based on human action (approve/reject/escalate). Demonstrates how HITL transitions from bolt-on approval gates…
In 14 listsDetails
Mem0
Mem0 is an intelligent memory layer for Large Language Models that enhances personalized AI experiences by retaining and utilizing contextual information across various applications. github | website | docs | discord | twitter | github profile | linkedin
In 13 listsDetails
Ray
A fast and simple framework for building and running distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library. ray.io
In 13 listsDetails
Haystack
Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search,…
In 13 listsDetails
PydanticAI
June 2026 harness-first redesign built around the Capability primitive: a single composable unit bundling instructions, tools, lifecycle hooks, and model settings. The split between a small stable core and a fast-moving pydantic-ai-harness lets capabilities graduate as they prove essential, while…
In 12 listsDetails