From Documents to Dialogues: Build a Custom AI Chatbot with RAG (Step-by-Step Tutorial)

Retrieval‑Augmented Generation (RAG) transforms static documents into interactive conversations by coupling a retrieval system with a generative language model. This tutorial walks you through every stage—from gathering source material to deploying a responsive chatbot—so you can create a domain‑specific assistant that cites its sources and stays up‑to‑date without retraining the model. By the end, you will have a working prototype that you can adapt for internal knowledge bases, customer support, or educational tools.

The guide assumes basic familiarity with Python and the command line, but each step includes concrete commands, version numbers, and cost estimates drawn from publicly available specifications and vendor documentation. All performance figures are sourced from benchmark reports, manufacturer datasheets, or independent lab measurements, ensuring you can replicate the setup with confidence.

Understanding Retrieval‑Augmented Generation and Its Advantages

RAG addresses a core limitation of large language models (LLMs): their knowledge is frozen at the time of training and they can hallucinate facts. By retrieving relevant passages from an external corpus at inference time, the model grounds its responses in verifiable data. According to a 2023 survey of 1,200 AI practitioners published in the Journal of Machine Learning Research, 78 % reported reduced hallucination rates when using a retrieval component alongside LLMs.

The retrieval stage typically relies on dense vector embeddings that capture semantic similarity. When a user query arrives, the system searches a vector store for the top‑k most relevant document chunks, concatenates them with the prompt, and feeds the combined input to the LLM. This two‑step process adds only a few hundred milliseconds of latency on modest hardware, as reported in the latency benchmarks released by the FAISS team in early 2024 (average query time = 210 ms for a 100 k‑document index on a CPU‑only setup).

From a cost perspective, RAG lets you leverage a smaller, cheaper LLM for generation while still achieving high‑quality answers because the retrieval supplies the necessary domain knowledge. For example, using the open‑source Llama‑2‑7B model with a retrieval layer can match the factual accuracy of GPT‑4‑Turbo on domain‑specific benchmarks while cutting token‑cost by roughly 80 %, according to a cost analysis published by Hugging Face in Q3 2023 (average cost per 1,000 tokens: $0.0002 for Llama‑2‑7B vs. $0.0010 for GPT‑4‑Turbo).

Preparing Your Data: Sourcing, Cleaning, and Chunking

The quality of the retrieval pipeline hinges on how well your source documents are processed. Begin by collecting the texts you want the chatbot to reference—PDFs, HTML pages, or plain‑text files. For a technical support bot, a common practice is to export the latest 500 KB of knowledge‑base articles from a Zendesk export, which yields roughly 120 documents averaging 4 KB each, based on the average article size reported in Zendesk’s 2023 usage statistics.

Next, clean the content: strip boilerplate headers, footers, and navigation menus; convert PDFs to text using pdfminer.six (version 2023.12.28) which preserves layout better than pure OCR for scanned documents; and normalize whitespace. According to the documentation of pdfminer.six, the library extracts text with an average fidelity of 96 % compared to manual transcription for PDFs under 10 pages.

Chunk the cleaned text into overlapping segments to preserve context across boundaries. A widely adopted strategy is to split on sentences and then group every 5 sentences with a 1‑sentence overlap, yielding chunks of roughly 150‑200 tokens. In a 2022 evaluation by the LangChain team, this configuration achieved a 4.2 % improvement in retrieval recall over fixed‑size 256‑token windows on the MS‑MARCO passage ranking dataset.

Store each chunk as a JSON line containing the text, a unique identifier, and metadata such as source file name and page number. This format enables fast ingestion into the vector store and simplifies later filtering by metadata.

Setting Up the Environment: Software, Hardware, and Cost Breakdown

You will need a Python 3.11 interpreter (the latest stable release as of October 2024) and a handful of libraries. The core stack includes:

• langchain==0.1.0 – the orchestration framework for RAG pipelines (released March 2023).
• sentence-transformers==2.2.2 – for generating dense embeddings (model: all-MiniLM-L6-v2).
• faiss-cpu==1.8.0 – the CPU‑only Faiss index for similarity search.
• openai==1.3.0 – optional, for accessing GPT‑4‑Turbo via the API.
• streamlit==1.32.0 – for building a quick web interface.

All of these packages are available on PyPI and install via pip install -r requirements.txt. The combined installation size is approximately 420 MB, according to the package metadata sizes listed on PyPI.

Hardware requirements depend on whether you run the embedding model on CPU or GPU. The all-MiniLM-L6-v2 model contains 22 M parameters and occupies about 90 MB of RAM. Running it on a modern laptop CPU (e.g., Intel i7‑12700H) yields an embedding throughput of ~350 sentences per second, as measured in the Sentence‑Transformers benchmark suite (2023). If you prefer GPU acceleration, a single NVIDIA RTX 3060 (6 GB GDDR6) provides roughly 2.1 k sentences per second, according to the GPU‑benchmarks published by NVIDIA in their “Deep Learning Performance Guide” (2024). The RTX 3060’s MSRP is $329, and its power draw averages 170 W under load, based on the manufacturer’s spec sheet.

For the generation step, you have two main options:

• Use OpenAI’s GPT‑4‑Turbo via the API. The service charges $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens (pricing page, effective Jan 2024). A typical exchange of 200 input tokens and 150 output tokens therefore costs about $0.009.
• Deploy Llama‑2‑7B locally with the transformers library. The model weights are 13.5 GB in FP16 format. Running it on an RTX 3060 yields an average latency of 420 ms per token generation step, according to the MLPerf‑Inference v3.1 results (2024). At this speed, a 150‑token response costs roughly $0.0015 in electricity (assuming $0.13/kWh and 170 W draw), far below the API price.

Our pick for a cost‑effective, private deployment is the Llama‑2‑7B on an RTX 3060, balancing latency, cost, and data privacy. If you require the highest possible fluency and are comfortable with API usage, GPT‑4‑Turbo remains the top‑ranked choice in the LMSYS Chatbot Arena leaderboard (March 2024), where it achieved an average Elo rating of 1240.

Building the Vector Store with FAISS

With your document chunks ready, the next step is to convert each chunk into a dense vector and index them for fast similarity search. Using Sentence‑Transformers, encode each chunk’s text:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(chunk_texts, show_progress_bar=True, batch_size=32)

The encoding process for a corpus of 10 k chunks (average 180 tokens each) completes in about 4 minutes on a CPU‑only i7‑12700H, as reported in the Sentence‑Transformers throughput benchmarks (2023). On an RTX 3060, the same task drops to ~45 seconds thanks to GPU acceleration.

Create a flat inner‑product FAISS index (appropriate for cosine similarity after L2‑normalizing the vectors):

import faiss
import numpy as np
dim = embeddings.shape[1]
index = faiss.IndexFlatIP(dim)
faiss.normalize_L2(embeddings)
index.add(embeddings)
faiss.write_index(index, 'doc_index.faiss')

The index file size approximates the number of vectors times the dimensionality times 4 bytes (float32). For 10 k vectors of 384 dimensions, the index occupies roughly 15 MB, matching the storage estimate provided in the FAISS README (2024).

At query time, encode the user question with the same model, normalize, and search the index for the top‑k vectors (commonly k = 4). The search latency for a 10 k‑vector index is under 1 ms on CPU and sub‑0.2 ms on GPU, according to the FAISS performance blog post (June 2023).

Integrating a Language Model: Choosing Between GPT‑4‑Turbo and Llama‑2‑7B

Now that you can retrieve relevant text snippets, you need a model that can generate a fluent answer conditioned on those snippets. The simplest approach is to construct a prompt that concatenates the retrieved passages, the user query, and a short instruction: “Answer the question using only the information provided below.”

If you opt for the OpenAI API, install the openai package and call the chat completion endpoint:

import openai
openai.api_key = 'sk-…'
response = openai.ChatCompletion.create(
    model='gpt-4-turbo',
    messages=[
        {"role": "system", "content": "You are a helpful assistant that answers based on the provided context."},
        {"role": "user", "content": f"Context: {retrieved_text}\n\nQuestion: {user_query}"}
    ],
    temperature=0.2
)
answer = response.choices[0].message['content']

According to OpenAI’s usage policy (updated Nov 2023), the model’s contextual window is 128 k tokens, ample for concatenating several retrieved chunks. The average latency for a single call with ~400 total tokens is 600 ms, as reported in the OpenAI API latency dashboard (Q2 2024).

For a self‑hosted solution, load Llama‑2‑7B with the transformers library and use torch for inference:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-chat-hf')
model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Llama-2-7b-chat-hf',
    torch_dtype=torch.float16,
    device_map='auto'
)

def generate_answer(context, query):
    prompt = f"""[INST] You are a helpful assistant. Use the context to answer the question.
Context: {context}
Question: {query} [/INST]"""
    inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
    output = model.generate(**inputs, max_new_tokens=200, temperature=0.2)
    return tokenizer.decode(output[0], skip_special_tokens=True)

Hugging Face’s model card for Llama‑2‑7B‑chat notes that the model was trained on 2 trillion tokens and optimized for dialogue. In the Open LLM Leaderboard (November 2023), Llama‑2‑7B‑chat scored an average of 5.3 / 7 on the MT‑Bench instruction‑following test, compared to 6.8 / 7 for GPT‑4‑Turbo.

When deploying locally, ensure you have at least 16 GB of system RAM to accommodate the model weights plus overhead; the transformers library reports a peak memory usage of ~14 GB on an RTX 3060 with FP16 weights, leaving room for the FAISS index and Python interpreter.

Implementing the Chatbot Interface with Streamlit

Written byTheo Grant

Theo Grant explores real-world AI applications, automation workflows, and hands-on tutorials at AI In Action Hub. Theo breaks down complex AI concepts into practical guides that help professionals and creators leverage AI in their daily work.