Let’s cut through the noise. Retrieval-Augmented Generation (RAG) isn’t just a buzzword; it’s the definitive architectural pattern for building modern, context-aware AI applications.
If you’ve spent any time working with Large Language Models (LLMs), you know they have a fatal flaw: they are confident liars. They hallucinate, they have strict training knowledge cutoffs, and they know absolutely nothing about your proprietary database, private codebases, or internal company wikis.
RAG bridges that gap. It takes your unstructured data—PDFs, Notion docs, Markdown files—and turns it into a searchable, semantic engine. It fetches exact, relevant context and feeds it directly into the LLM’s brain just in time to answer a user’s question.
The Core Process: How RAG Actually Works
Before we look at the implementation, let’s map the architecture. At its heart, RAG is a two-step dance: an indexing pipeline and a retrieval pipeline.
Architecture 1. Ingestion and Indexing You don’t just dump a 100-page PDF into an LLM. You parse it and split it into manageable “chunks” (usually a few hundred tokens each) using something like a recursive character text splitter to ensure you don’t cut paragraphs in half.
Next, you pass these chunks through an Embedding Model (like OpenAI’s text-embedding-3-small) . This model converts human text into a dense vector—a massive array of floating-point numbers representing the semantic meaning of the text. We store these vectors in a Vector Database (like Pinecone, Qdrant, or a pgvector extension).
2. Retrieval and Generation
When a user asks, “What is our company’s refund policy?”, we embed that query into a vector too. The Vector DB performs a similarity search—calculating the mathematical distance between vectors using formulas like Cosine Similarity:
It returns the top 5 most mathematically relevant text chunks. We then shove those chunks into the system prompt, look at the LLM, and say: “Answer the user’s question using strictly the following context.”
The Evolution of RAG Topologies
The ecosystem has rapidly evolved beyond simple similarity search. Today, we categorise RAG into several advanced architectures:
- Standard/Naive RAG: The foundational retrieval and generation pipeline.
- Corrective RAG (CRAG): Introduces a retrieval evaluator to assess the quality of fetched documents, triggering web searches if internal confidence is low.
- Speculative RAG: Predicts user intent and drafts multiple potential response pathways simultaneously to reduce latency.
- Fusion RAG (RAG-Fusion): Generates multiple variations of the user’s query, retrieves documents for all of them, and applies Reciprocal Rank Fusion to surface the absolute best context.
- Agentic RAG: Uses LLMs as reasoning agents to determine if they need to retrieve data, where to get it from, and iteratively query multiple tools until the user’s complex problem is solved.
- Graph RAG: Combines Vector DBs with Knowledge Graphs to understand complex entity relationships, solving the “scattered context” problem.
How to Build It: Choosing Your Stack
When it comes to building a RAG pipeline in Python, you generally have three paths: going bare-metal with the raw SDKs, using an orchestration framework like LangChain or use a pre-packaged library like OpenRAG.
Route 1: Close to the Metal (OpenAI & Claude SDKs)
Writing RAG entirely from scratch using the raw SDKs gives you absolute control over your token limits, error handling, and data flow. You aren’t fighting a framework; you’re just writing Python.
Typically, you use the OpenAI SDK to generate your embeddings. Once you handle the vector math and retrieve your chunks, you format the prompt yourself.
If you are using the Anthropic Claude SDK for the generation step, the code is beautifully explicit. You handle the message loop manually:
import anthropic client = anthropic.Anthropic(api_key="your_api_key") # You write the logic to hit your Vector DB and return strings retrieved_context = get_vector_search_results(user_query) system_prompt = f"You are a helpful assistant. Answer based ONLY on this context: {retrieved_context}" response = client.messages.create( model="claude-3-5-sonnet-latest", max_tokens=1024, system=system_prompt, messages=[ {"role": "user", "content": user_query} ] ) print(response.content[0].text) By using the raw Claude SDK, you can manually implement Prompt Caching . If you are constantly passing the same massive 50-page company handbook into the context window, you can flag that block of text to be cached natively via the API. Claude stores it, and your API costs drop by up to 90% for subsequent queries. Frameworks often obscure these low-level, money-saving features.
Route 2: The Orchestrator (LangChain)
If you don’t want to write the repetitive glue code for parsing PDFs, chunking text, and formatting prompts, LangChain is the heavy hitter in the space.
LangChain abstracts the entire RAG pipeline into composable blocks. You don’t have to write custom vector search logic; you just declare a retriever and pipe it together using LCEL (LangChain Expression Language).
Here is what a dense, modern LangChain pipeline looks like using OpenAI for both embeddings and generation:
from langchain_openai import OpenAIEmbeddings, ChatOpenAI from langchain_community.vectorstores import FAISS from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough # 1. Set up the LLM and Embeddings via OpenAI llm = ChatOpenAI(model="gpt-4o") embeddings = OpenAIEmbeddings(model="text-embedding-3-small") # 2. Assume we loaded and chunked documents earlier vectorstore = FAISS.from_documents(chunks, embeddings) retriever = vectorstore.as_retriever(search_kwargs={"k": 5}) # 3. Create the prompt template template = """Answer the question based only on the following context: {context} Question: {question}""" prompt = ChatPromptTemplate.from_template(template) # 4. Chain it all together using LCEL rag_chain = ( {"context": retriever, "question": RunnablePassthrough()} | prompt | llm ) # Run it response = rag_chain.invoke("What is the refund policy?") print(response.content) LangChain handles the heavy lifting. It runs the user query through the embedding model, queries FAISS (your local vector store), grabs the top 5 chunks, formats the string, calls OpenAI, and returns the parsed output—all cleanly executed in that rag_chain pipe.
The Trade-off: LangChain is highly opinionated. When things go wrong, debugging a massive abstract chain of runnables can be incredibly frustrating. If you need tight, granular control over your architecture, the abstraction can feel like a straitjacket. But for rapid prototyping and standardised workflows, it is unbeatable.
Route 3: OpenRAG ( Pre-Packaged Stack)
You can take over your business problem immediately and implement a production-ready RAG architecture without the headache using OpenRAG
The Trade-off
You might worry that adopting a pre-packaged stack limits your flexibility. But if you truly understand the intricacies of vector search and generative orchestration, you know that building a basic RAG app takes a weekend, while building a production-grade RAG app takes months of finessing.
OpenRAG is an opinionated, open-source stack that does the heavy lifting for you. It packages everything you need into a single platform, giving you the fast time-to-market of a managed service with the zero vendor lock-in of an open-source project.
Under the Hood
What makes a pipeline like this different? It’s the time the community spent optimising the unglamorous parts of the data pipeline. It’s like sinking 100+ hours into GTA—you stop looking at the map because you know the exact layout of every back alley.
Here is a look at the architecture OpenRAG employs:
- Docling for Intelligent Ingestion: It doesn’t just use blind, fixed-character chunking. It leverages Docling to handle messy, real-world data with intelligent semantic parsing, retaining document structure and tabular data integrity.
- OpenSearch for Retrieval: Instead of a basic vector store, it utilizes OpenSearch for production-grade performance, allowing you to combine dense vector search (for semantic meaning) with traditional search at a massive scale.
- Langflow for Orchestration: The real magic is the drag-and-drop workflow builder. You get a visual interface powered by Langflow for rapid iteration, agentic multi-tool coordination, and dynamic re-ranking to filter out noise and combat context-window dilution.
- Ready to Run: It ships with a React frontend (Next.js) and a FastAPI backend hooked up out of the box.
What’s Next for RAG?
Global adoption across multiple modalities. Right now, RAG is heavily mistreated and under utilised as just a backend for “Chatbots.” It holds vastly more capability than that.
The industry is continuously working to make these architectures faster while providing an unparalleled UX. Moving forward, the focus isn’t just on PDFs—it’s on live, dynamic integrations. We are looking at pipelines that securely ingest and ground models using structured SQL Databases, Salesforce CRMs, Slack, Microsoft Teams, and other synchronous communication platforms. The future of RAG is ubiquitous enterprise intelligence.
Phil Nash breaks down the OpenRAG stack
This presentation provides an excellent technical overview of how OpenRAG combines Docling, OpenSearch, and Langflow to create a robust, production-ready retrieval architecture.
Final Thoughts
It’s an incredible time to be building software. The tools whether you choose the granular control of the Claude and OpenAI SDKs or the rapid orchestration of LangChain are right there for the taking.
Happy Coding :)
