Retrieval-Augmented Generation (RAG)
Definition
Retrieval-Augmented Generation (RAG) is an architecture that improves large language model (LLM) answers by retrieving relevant documents from an external knowledge base at query time and injecting them into the model’s context. Instead of relying only on what the model memorized during training, a RAG system searches your own data — wikis, PDFs, tickets, code — and grounds the answer in those sources, which reduces hallucination and keeps answers current without retraining.
Why it matters for AI jobs
RAG is the single most common production LLM architecture in industry. Nearly every enterprise AI assistant, internal knowledge bot, and "chat with your docs" product is a RAG system. Job postings for AI engineers and forward-deployed engineers at companies like OpenAI, Anthropic, and Cohere routinely list production RAG experience as a core requirement, because it exercises the full stack: data pipelines, embeddings, vector databases, prompt construction, and evaluation.
Key concepts
- Chunking — splitting documents into retrievable passages; chunk size and overlap strongly affect quality.
- Embeddings — dense vector representations of text used for semantic similarity search.
- Vector search — approximate nearest-neighbor (ANN) lookup in a vector store such as pgvector, Pinecone, or Qdrant.
- Hybrid search — combining vector similarity with keyword (BM25) search for better recall.
- Reranking — a second-stage model (e.g. Cohere Rerank, cross-encoders) that reorders retrieved passages by true relevance.
- Grounded generation & citations — prompting the LLM to answer only from retrieved context and cite sources.
- RAG evaluation — measuring retrieval recall, answer faithfulness, and relevance (e.g. with Ragas or custom evals).
Learning path
- Build a minimal RAG pipeline by hand (no framework): load documents, chunk them, embed with an embedding API, store vectors in Postgres + pgvector, retrieve top-k, and prompt an LLM with the results.
- Add hybrid search (BM25 + vector) and a reranker, then measure how retrieval quality changes on a small labeled question set.
- Add evaluation: build 30–50 question–answer pairs from your corpus and track retrieval hit-rate and answer faithfulness on every change.
- Productionize: incremental indexing, metadata filters, access control, caching, and latency budgets.
- Ship a portfolio project, e.g. "chat with my résumé/wiki" deployed on a cloud platform with monitoring.