Haystack is an open-source NLP and LLM framework by deepset. It provides a Pipeline abstraction for composing document processing, retrieval, and generation components into end-to-end RAG (Retrieval-Augmented Generation) and agent systems. Haystack 2.0 (released 2024) introduced a redesigned component-based pipeline replacing the legacy v1 Node system.

What document stores does Haystack support?

Haystack has document store integrations for Elasticsearch, OpenSearch, Qdrant, Weaviate, Pinecone, Chroma, Milvus, and an in-memory InMemoryDocumentStore for development. Each store integrates with Haystack's EmbeddingRetriever and BM25Retriever via a consistent interface, so you can swap stores without changing pipeline code.

Hayhooks is Haystack's REST API server. You define a Pipeline in YAML or Python, point Hayhooks at it, and it exposes the pipeline as an HTTP endpoint. This lets you deploy RAG pipelines as a service without writing custom FastAPI or Flask boilerplate. It also serves a /docs OpenAPI spec automatically.

How does Haystack compare to LangChain and LlamaIndex?

All three build LLM pipelines in Python. LangChain has the largest ecosystem but historically complex abstractions. LlamaIndex focuses on data ingestion and RAG for structured/unstructured documents. Haystack has the cleanest pipeline model for search-centric use cases and is the most popular choice in enterprise NLP teams already using Elasticsearch or OpenSearch.

Haystack RAG 9 min read

Haystack Guide 2025: Open-Source NLP Framework for RAG & Agents

Haystack by deepset is the go-to Python framework for teams building search and RAG systems on top of Elasticsearch, Qdrant, or Weaviate. This guide covers the 2.0 Pipeline model, a working RAG example, and how it compares to LangChain.

What Is Haystack?

Haystack is an open-source framework by deepset for building production-grade NLP and LLM applications. Where LangChain provides a broad toolkit for any LLM use case, Haystack specializes in the search-and-retrieval part of the stack — making it particularly strong for RAG pipelines, question answering over documents, and enterprise search.

Haystack 2.0 (released early 2024) is a ground-up redesign. The new Pipeline model is cleaner and more composable than the legacy v1 Node/Graph system.

pip install haystack-ai

Core Components

A Haystack pipeline is a directed graph of components connected by named inputs and outputs:

Component type	Examples	What it does
Document Converters	PyPDFToDocument, HTMLToDocument	Convert files to Document objects
Preprocessors	DocumentSplitter, DocumentCleaner	Chunk and clean documents
Embedders	OpenAITextEmbedder, SentenceTransformersEmbedder	Convert text to vectors
Document Stores	QdrantDocumentStore, ChromaDocumentStore	Store and index documents
Retrievers	QdrantEmbeddingRetriever, BM25Retriever	Retrieve relevant documents
Generators	OpenAIGenerator, AnthropicGenerator, OllamaGenerator	Call LLMs and return answers
Routers	MetadataRouter, ConditionalRouter	Route documents by condition

Supported Document Stores & LLMs

Haystack's integrations are installed as separate packages:

Install document store integrations

pip install qdrant-haystack          # Qdrant
pip install weaviate-haystack        # Weaviate
pip install elasticsearch-haystack   # Elasticsearch / OpenSearch
pip install chroma-haystack          # Chroma
pip install pinecone-haystack        # Pinecone

Supported LLM generators include OpenAIGenerator, AnthropicGenerator, HuggingFaceAPIGenerator, OllamaGenerator, and CohereGenerator.

Building a RAG Pipeline

A minimal end-to-end RAG pipeline: PDF input → split → embed → store → retrieve → generate answer.

rag_pipeline.py

from haystack import Pipeline
from haystack.components.converters import PyPDFToDocument
from haystack.components.preprocessors import DocumentSplitter, DocumentCleaner
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.writers import DocumentWriter
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.document_stores.in_memory import InMemoryDocumentStore
import os

os.environ["OPENAI_API_KEY"] = "sk-..."

# ---------- Indexing pipeline ----------
store = InMemoryDocumentStore()

index_pipeline = Pipeline()
index_pipeline.add_component("converter", PyPDFToDocument())
index_pipeline.add_component("cleaner",   DocumentCleaner())
index_pipeline.add_component("splitter",  DocumentSplitter(split_by="word", split_length=200))
index_pipeline.add_component("embedder",  OpenAIDocumentEmbedder())
index_pipeline.add_component("writer",    DocumentWriter(document_store=store))

index_pipeline.connect("converter", "cleaner")
index_pipeline.connect("cleaner",   "splitter")
index_pipeline.connect("splitter",  "embedder")
index_pipeline.connect("embedder",  "writer")

index_pipeline.run({"converter": {"sources": ["my_docs.pdf"]}})

# ---------- Query pipeline ----------
template = """
Given the context below, answer the question.

Context:
{% for doc in documents %}
  {{ doc.content }}
{% endfor %}

Question: {{ question }}
Answer:
"""

query_pipeline = Pipeline()
query_pipeline.add_component("embedder",   OpenAITextEmbedder())
query_pipeline.add_component("retriever",  InMemoryEmbeddingRetriever(document_store=store, top_k=3))
query_pipeline.add_component("builder",    PromptBuilder(template=template))
query_pipeline.add_component("generator",  OpenAIGenerator(model="gpt-4o-mini"))

query_pipeline.connect("embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever",          "builder.documents")
query_pipeline.connect("builder",            "generator")

result = query_pipeline.run({
    "embedder": {"text": "What is the refund policy?"},
    "builder":  {"question": "What is the refund policy?"}
})

print(result["generator"]["replies"][0])

Haystack Agents

Haystack 2.x supports tool-calling agents via OpenAIChatGenerator with tools. The agent loop is implemented as a Pipeline with a conditional router that re-runs until the LLM returns a final answer:

agent_example.py (simplified)

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import Tool

def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny, 22°C."

weather_tool = Tool(
    name="get_weather",
    description="Get current weather for a city",
    parameters={"city": {"type": "string"}},
    function=get_weather,
)

generator = OpenAIChatGenerator(model="gpt-4o-mini", tools=[weather_tool])

messages = [ChatMessage.from_user("What's the weather in Berlin?")]
response = generator.run(messages=messages)
print(response["replies"][0].text)

Hayhooks: REST API for Pipelines

Hayhooks wraps any Haystack Pipeline in a FastAPI REST server. Define your pipeline, deploy it, and call it via HTTP — no custom server code required:

pip install hayhooks

# Deploy a pipeline YAML
hayhooks run --pipelines-dir ./my_pipelines/

# Pipeline is now available at:
# POST http://localhost:1416/draw   (visualize)
# POST http://localhost:1416/query_pipeline/run

Haystack vs LangChain vs LlamaIndex

Framework	Pipeline model	Search focus	Best for
Haystack	Typed DAG components	Very high	Enterprise search, RAG, NLP
LangChain	LCEL chains + agents	Medium	General LLM apps, large ecosystem
LlamaIndex	Query engines + index	High	Document Q&A, structured data RAG

See also: LangChain guide · LlamaIndex guide

Monitor Your RAG Backend APIs

Haystack pipelines depend on OpenAI or Anthropic generators and vector store APIs like Qdrant or Weaviate. Prismix tracks the uptime of all of them so you know immediately when your RAG pipeline is at risk.

Check API Status →