Prismix
LlamaIndex RAG 10 min read

LlamaIndex Guide 2025: Build RAG Apps and AI Agents

A practical guide for developers building RAG pipelines and AI agents — core concepts, Python quickstart, vector store integrations, LLM provider support, and a comparison with LangChain.

What Is LlamaIndex?

LlamaIndex is a Python data framework that connects LLMs to your own data. Where ChatGPT or Claude only knows their training data, LlamaIndex lets you build apps that answer questions about your PDFs, databases, APIs, and files.

The core use case is RAG (Retrieval-Augmented Generation): ingest your documents into a vector index, retrieve the relevant chunks when a user asks a question, and pass those chunks to an LLM to generate a cited answer.

LlamaIndex also powers AI agents — LLMs that reason over multiple data sources, call tools, and take multi-step actions to complete complex tasks.

Core Concepts

Documents & Nodes

A Document wraps raw content (PDF page, web page, database row) with metadata. A Node is a chunk of a Document — the actual unit stored in the vector index. Default chunk size: 1024 tokens with 20-token overlap.

Index

A data structure over your Nodes. VectorStoreIndex embeds each Node and stores it for semantic search. Other index types: SummaryIndex, KnowledgeGraphIndex, KeywordTableIndex.

Query Engine

Takes a natural language query, retrieves relevant Nodes (top-k=2 by default), and passes them to the LLM along with the question. Returns a Response object with the answer and source citations.

Response Synthesizer

Controls how retrieved nodes are combined and passed to the LLM. Modes: compact (default, fits all context in one LLM call), refine (iterative, better for large context), tree_summarize (hierarchical summarization).

Python Quickstart (5 Minutes)

# Install

pip install llama-index openai

# RAG over a folder of PDFs/text files

import os
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Set your OpenAI API key (or use another LLM — see below)
os.environ["OPENAI_API_KEY"] = "sk-..."

# 1. Load documents from a folder (supports PDF, TXT, DOCX, HTML, CSV...)
documents = SimpleDirectoryReader("./data").load_data()

# 2. Build a vector index (embeds each chunk, stores in memory)
index = VectorStoreIndex.from_documents(documents)

# 3. Create a query engine
query_engine = index.as_query_engine()

# 4. Query
response = query_engine.query("What are the main conclusions in the report?")
print(response)  # Answer + sources

# Access source nodes:
for node in response.source_nodes:
    print(f"Source: {node.metadata.get('file_name', 'unknown')}")
    print(f"Score: {node.score:.3f}")
    print(f"Text: {node.text[:200]}...")

Tip: The in-memory index doesn't persist. To save and reload: index.storage_context.persist("./storage") then load_index_from_storage(StorageContext.from_defaults(persist_dir="./storage")).

Supported LLMs

LlamaIndex works with any major LLM provider via optional packages: 

# OpenAI (default)
pip install llama-index-llms-openai
from llama_index.llms.openai import OpenAI
llm = OpenAI(model="gpt-4o", api_key="sk-...")

# Anthropic Claude
pip install llama-index-llms-anthropic
from llama_index.llms.anthropic import Anthropic
llm = Anthropic(model="claude-sonnet-4-6", api_key="sk-ant-...")

# Groq (fast inference)
pip install llama-index-llms-groq
from llama_index.llms.groq import Groq
llm = Groq(model="llama-3.3-70b-versatile", api_key="gsk_...")

# Ollama (local)
pip install llama-index-llms-ollama
from llama_index.llms.ollama import Ollama
llm = Ollama(model="llama3.2", request_timeout=120.0)

# Set as global default:
from llama_index.core import Settings
Settings.llm = llm

Vector Store Integrations

Store Type Free tier Best for
Chroma Local / OSS ✅ Unlimited local Dev / prototyping
Pinecone Managed cloud ✅ 1M vectors Production scale
Weaviate Hybrid cloud/OSS ✅ Cloud sandbox BM25 + vector hybrid
Qdrant Local / cloud ✅ Local free High-performance, Rust
pgvector PostgreSQL ext ✅ Self-hosted Existing Postgres stack

# Example: persist to Chroma

pip install llama-index-vector-stores-chroma chromadb

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

LlamaIndex vs LangChain

Factor LlamaIndex LangChain
Primary focus RAG + data ingestion Agents + orchestration
Ease of use Simpler RAG API More boilerplate but flexible
GitHub stars 36k+ 90k+
Integrations 160+ (data/vector) 600+ (broad ecosystem)
Multi-agent Workflows API LangGraph (dedicated)
Choose when Document Q&A, RAG apps Complex agent pipelines

For a deeper comparison, see LangChain guide. Many production teams use LlamaIndex for data ingestion and LangChain for the agent layer on top.

Monitor Your LLM Providers

LlamaIndex apps depend on OpenAI, Anthropic, Groq, or other providers. When those APIs go down, your RAG pipeline fails silently. Prismix monitors all major LLM providers and alerts you instantly on outages.

Monitor AI APIs Free →