- Step 1: Define the Chatbot’s Scope, Data Sources, and Success Metrics
- Step 2: Prepare and Chunk the Source Corpus
- Step 3: Choose and Set Up the Vector Store
- Step 4: Select the Language Model and Configure the Generation Pipeline
- Step 5: Build the Retrieval‑Augmented Generation Chain
- Step 6: Deploy the Chatbot as a Simple API
🔧 Build a Custom AI Chatbot Using RAG: A 7-Step Hands-On Tutorial
By the end of this guide you will be able to design, assemble, and deploy a retrieval‑augmented generation (RAG) chatbot that answers questions over your own private data. The workflow covers everything from defining the bot’s purpose to setting up a lightweight vector store, connecting a large language model (LLM), and exposing the service through a simple API. All steps use openly available tools and cloud‑friendly configurations, so you can replicate the prototype on a laptop or a modest‑cost virtual machine.
Each step includes concrete numbers — chunk sizes, storage footprints, instance pricing, and latency estimates — drawn from publicly published specifications, vendor documentation, and independent benchmark reports. Follow the sequence, adjust the parameters to your use case, and you’ll have a functional RAG prototype ready for further iteration.
Define the Chatbot’s Scope, Data Sources, and Success Metrics
Start by articulating a narrow, measurable goal. For example, “Answer employee questions about internal HR policies” or “Provide troubleshooting steps for a specific software product.” A focused scope reduces the volume of data you need to ingest and makes evaluation easier. According to a 2024 survey of 420 developers conducted by the AI Engineering Consortium, projects that limited their initial knowledge base to fewer than 200 k tokens reported a 32% faster time‑to‑demo than those that started with corpora over 1 M tokens (source: AI Engineering Consortium Report, 2024).
List the exact documents you will use: PDFs, HTML pages, or CSV files. Record their formats, average size, and any licensing constraints. If you plan to scrape a public website, note the allowed request rate from the site’s robots.txt to avoid legal issues. For internal documents, estimate the total token count using a tokenizer such as GPT‑2’s BPE; a typical 10‑page PDF contains roughly 750 tokens, so a collection of 150 files yields about 112 k tokens.
Define success metrics that you can measure without building a full‑scale user study. Common proxies include retrieval recall@5 (the percentage of queries where the correct answer appears in the top five retrieved chunks) and generation correctness (rated on a 1‑5 scale by a subject‑matter expert). Published baselines show that a well‑tuned RAG pipeline can achieve recall@5 of 0.78 on domain‑specific FAQs when using a cosine similarity search over sentence‑transformer embeddings (source: Lee et al., “Evaluating Retrieval‑Augmented Generation,” ACL 2023). Set a target of at least 0.75 recall@5 before moving to the next step.
Prepare and Chunk the Source Corpus
Transform raw documents into uniform text chunks that the embedding model can process. A widely adopted approach is recursive character splitting with overlap, which preserves context at chunk boundaries. For most English‑language texts, a chunk size of 512 tokens with a 50‑token overlap provides a good balance between granularity and coherence, according to the LangChain documentation (v0.2.0, 2024) that cites internal benchmarks on the MTEB retrieval leaderboard.
Apply the splitter using the RecursiveCharacterTextSplitter class from LangChain. Example configuration in Python:
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
length_function=len,
is_separator_regex=False,
)
Run the splitter on each file and store the resulting chunks in a list. For a corpus of 150 PDFs averaging 10 pages each, you will generate roughly 300 chunks per file, yielding about 45 000 chunks total. Each chunk, when encoded with the sentence-transformers/all-MiniLM-L6-v2 model, produces a 384‑dimensional vector.
Estimate the storage footprint of the raw text and the vectors. The raw chunks occupy approximately 22 MB (assuming ~5 bytes per character). The FAISS index that holds the vectors uses 4 bytes per float32 component, so 384 × 4 = 1 536 bytes per vector, plus a small overhead for the index structure. For 45 000 vectors, the index size is about 70 MB (source: FAISS GitHub wiki, “Index size estimation”). This fits comfortably in RAM on a modest instance, allowing rapid similarity searches without disk I/O.
Choose and Set Up the Vector Store
Our pick for the vector database in a prototype setting is FAISS (Facebook AI Similarity Search) because it offers sub‑millisecond query latency on CPU and requires no external service. Independent benchmarking by the ANN‑Survey 2023 ranked FAISS among the top three libraries for recall‑speed trade‑off on datasets under 1 M vectors, with a median query time of 0.4 ms on an Intel Xeon E5‑2680 v4 (source: ANN‑Survey 2023, Table 4).
Initialize a flat index (IndexFlatIP) for exact inner‑product search, which works well when embeddings are L2‑normalized. Normalize the MiniLM embeddings to unit length before adding them; this converts inner product to cosine similarity. The following snippet builds the index:
import faiss, numpy as np
embeddings = np.load('embeddings.npy').astype('float32')
faiss.normalize_L2(embeddings)
index = faiss.IndexFlatIP(embeddings.shape[1])
index.add(embeddings)
faiss.write_index(index, 'faiss_index.bin')
If you anticipate scaling beyond a few hundred thousand vectors, consider switching to an IVF‑PQ index (inverted file with product quantization) to reduce RAM usage. The same ANN‑Survey reports that an IVF‑PQ with 1024 centroids and 8‑bit codebooks yields a 4× memory reduction with less than 5% recall loss at the same query throughput.
Estimate the cost of running the vector store on a cloud VM. A t3.medium instance (2 vCPU, 4 GB RAM) on AWS costs $0.0416 per hour (on‑demand, us‑east‑1, September 2025). With the index residing in memory, the instance can handle roughly 150 queries per second before CPU utilization exceeds 70%, based on the latency figures above and a simple queuing model (M/M/1). This translates to a monthly cost of about $30 if the bot runs continuously for 8 hours per day.
Select the Language Model and Configure the Generation Pipeline
For the generative component, we recommend using OpenAI’s GPT‑4‑turbo model via the API, as it provides the best trade‑off between reasoning ability and cost for most enterprise use cases. According to OpenAI’s pricing page (accessed September 2025), GPT‑4‑turbo charges $0.03 per 1 000 input tokens and $0.06 per 1 000 output tokens. A typical query that retrieves three 512‑token chunks (≈1 500 tokens) and generates a 150‑token answer consumes roughly 1 650 input tokens and 150 output tokens, resulting in a cost of about $0.0059 per interaction.
If you prefer an open‑source alternative to avoid per‑token fees, the Mistral‑7B‑Instruct v0.2 model, quantized to 4‑bit using GGUF, runs comfortably on a single GPU with 16 GB VRAM. Independent benchmarks from Hugging Face’s Open LLM Leaderboard (June 2024) show that the 4‑bit Mistral‑7B achieves an average MT‑Bench score of 6.2, comparable to GPT‑3.5‑turbo, while consuming roughly 0.9 GB of GPU memory per batch of size 1.
Set up the generation wrapper using LangChain’s ChatOpenAI or HuggingFacePipeline class. Define a prompt template that instructs the model to answer strictly based on the retrieved context, for example:
template = """Answer the question using only the information below.
If the answer cannot be derived from the context, say you don't know.
Context:
{context}
Question:
{question}
Answer:
"""
This mitigates hallucination, a concern highlighted in a 2023 study by Liu et al., which found that unrestricted LLMs produced false statements in 27% of fact‑checking prompts, whereas retrieval‑conditioned generation reduced the error rate to 9% (source: Liu et al., “Hallucination in Large Language Models,” EMNLP 2023).
Build the Retrieval‑Augmented Generation Chain
Connect the vector store retriever to the LLM prompt template to form a RAG chain. In LangChain, this is accomplished with the RetrievalQA utility, which internally performs a similarity search, formats the retrieved documents into the prompt, and calls the language model.
from langchain.chains import RetrievalQA
from langchain.vectorstores import FAISS
from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vectorstore = FAISS.load_local("faiss_index.bin", embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
qa_chain = RetrievalQA.from_chain_type(
llm=ChatOpenAI(model_name="gpt-4-turbo", temperature=0),
chain_type="stuff",
retriever=retriever,
return_source_documents=True,
)
The k=3 parameter tells the retriever to fetch the three most similar chunks, which aligns with the token budget used in the cost estimate above. The stuff chain type concatenates all retrieved documents into a single prompt; for longer contexts you could switch to map_reduce or refine to avoid exceeding the model’s maximum context length (8 192 tokens for GPT‑4‑turbo).
Estimate the end‑to‑end latency for a single query. The embedding step for the user question adds roughly 2 ms (MiniLM on CPU). The FAISS similarity search for 3‑nearest neighbors takes about 0.4 ms per vector, or ~1.2 ms total. Prompt construction and API overhead contribute another 50 ms, while the GPT‑4‑turbo inference latency averages 600 ms for a 150‑token output (based on OpenAI’s published median latency for the turbo endpoint, Q3 2024). Summing these yields an approximate total response time of 660 ms, well under the one‑second threshold often cited for satisfactory conversational flow (source: Nielsen Norman Group, “Response Times in Conversational Interfaces,” 2022).
Deploy the Chatbot as a Simple API
Expose the qa_chain through a lightweight REST endpoint using FastAPI, which adds minimal overhead and integrates smoothly with asynchronous Uvicorn workers. A minimal implementation looks like this:
from fastapi import FastAPI, Body
app = FastAPI()@app.post("/ask")
async def ask(question: str = Body(..., embed=True)):
result = qa_chain({"query": question})
return {"answer": result["result"], "sources": [doc.page_content for doc inGet the AI Edge, Weekly
The tools, tutorials, and trends that actually pay — no hype.
