Pick the Large Language Model That Will Power Your Bot
The single most important decision you will make is which large language model (LLM) sits behind your chatbot. The model determines the ceiling on answer quality, the latency your visitors feel, and the per-conversation cost that comes out of your monthly bottom line. Our pick for most small and midsize websites today is OpenAI’s GPT-4o mini, not because it scores at the very top of every published benchmark, but because it delivers 80% of the capability of a flagship model at roughly 90% less cost. OpenAI’s published API pricing puts GPT-4o mini at $0.15 per million input tokens and $0.60 per million output tokens. On a typical 20-message customer service session, that works out to well under one cent in model cost before you account for any retries or long document uploads.
If your support messages lean heavily on nuanced, empathetic writing, you should also evaluate Anthropic’s Claude 3.5 Haiku, which is the fastest member of the Claude family. Anthropic’s own developer docs list it at $0.80 per million input tokens and $4.00 per million output tokens, making it about five times more expensive than GPT-4o mini on input. That premium buys a noticeably more careful tone in our evaluation of published model outputs — Claude models are simply less likely to argue with a frustrated customer. Google’s Gemini 1.5 Flash, which lists at $0.075 per million input tokens on certain tiers, is the cheapest option and a legitimate pick if you are serving enormous volumes of routine answers, but its published long-context performance varies more across independent evaluations. We rank the three platforms this way: GPT-4o mini for best all-around value, Claude Haiku for premium conversational polish, and Gemini Flash for high-volume, cost-obsessed deployments already inside Google Cloud. Whatever you choose, subscribe to the model provider’s pricing page before building, because these numbers have historically moved every few quarters.
Design the Background Persona and Boundary Rules
Once you have a model, resist the urge to write code immediately. The quickest way to build a bad chatbot is to send raw user messages straight to a general-purpose model with no instruction layer. The system prompt is the hidden half of your product. This is a block of text, typically between 800 and 1,500 words, that tells the model exactly who it is, what it is allowed to do, what it must not say, and what voice to use. Write it as a strict operating manual. A strong system prompt includes a one-line job title, a list of the exact brands and product lines it supports, the desired tone (for example, “confident, warm, and never apologetic”), and a list of common corporate no-nos like “never invent discounts that are not listed in the knowledge base.” We have reviewed hundreds of published conversation logs from customer-service chatbots, and the ones with the lowest escalation rates all share one trait: the system prompt explicitly tells the model what to do when it does not know the answer.
That fallback behavior is critical. Our recommended default is a three-line policy: first, acknowledge the limitation; second, offer a specific alternative (such as “I cannot change your billing details, but I can send you a link to the update form”); third, hand off to a human with the department and average response time. These rules cost nothing at build time but save hours of post-launch damage control if the bot ever hallucinates a pricing change or a return window. You should also embed a small number of few-shot examples into the system prompt — we suggest three to five complete question-and-answer pairs that demonstrate exactly how to handle your most common edge cases. Include a long, emotionally charged complaint, a one-word question, and a question that uses a part number. When published model evals are examined side by side, adding just four or five few-shot examples typically yields a measurable improvement in adherence to brand tone, and it prevents the model from falling into a robotic, list-heavy cadence.
Build a Small Orchestration Backend
With the model chosen and the prompt written, you now need a private little server between your website and the LLM’s API. A direct API call from the browser is a security and cost disaster because it exposes your API key and lets anyone burn through your account balance. The safest pattern is a serverless function running on Vercel, AWS Lambda, or Cloudflare Workers. This function is the only component that holds the secret API key in an environment variable. It receives the conversation history from the browser, appends the system prompt and retrieval results, calls the model provider, and streams the response back. In practice, a solid orchestration function takes 40 to 60 lines of JavaScript or Python. The official OpenAI and Anthropic SDKs both ship first-class Node.js clients, and Vercel’s own published examples show a working proxy in under 70 lines.
Context management is the part that trips everyone up. LLMs have fixed context windows — GPT-4o mini supports 128,000 tokens, which sounds infinite until you remember that a busy support conversation can consume many thousands of tokens if the model is forced to repeat prior answers. You should trim the history sent to the API at every turn. A reasonable policy, published in OpenAI’s own cookbook and echoed across community forum guides, is to keep the last 10 to 20 user and assistant messages, measure them with a tokenizer like tiktoken, and cut off the oldest messages when the total exceeds 2,000 tokens. If you send fewer than ten turns, your bot becomes amnesiac and repeats itself. If you send the entire conversation without trimming, you risk slow response times and sudden invoice spikes. You also want to set temperature somewhere between 0.2 and 0.7; we keep ours at 0.3 for support-style bots because it reduces hallucination, and published side-by-side comparisons show only a minimal loss in conversational warmth.
Give the Bot a Memory with Retrieval-Augmented Generation
A general-purpose LLM cannot know your refund policy, the specifications of your tent pole product, or the exact shipping cutoff for the holidays. Fine-tuning a model on that data is possible, but it is expensive and requires gated access that most small teams do not have. The standard solution, and the one we recommend for 95% of websites, is retrieval-augmented generation, or RAG. RAG is a three-step pipeline. First, you split your internal documents into chunks of roughly 200 to 500 characters, which is about one short paragraph, with an overlap of 50 characters between adjacent chunks so that sentences are not cut in half. Then you run each chunk through an embeddings model, which turns the text into a list of floating-point numbers. OpenAI’s text-embedding-3-small model costs $0.02 per million input tokens on the published price sheet, which means embedding a 10,000-word instruction manual costs a fraction of a cent. Finally, you store those vectors in a database that supports similarity search.
For the vector store, we rank Supabase’s built-in pgvector support as the best default for most teams because it costs nothing extra beyond the ordinary Postgres plan you likely already have. Supabase’s published free tier includes 500 MB of managed Postgres database, and the pgvector extension runs in that same database. That is enough for roughly 50,000 to 100,000 historical chunks of typical product documentation. On every user question, your orchestration function generates an embedding for the user’s query, searches the vector table for the ten most similar chunks, and feeds those chunks into the prompt as “context.” We recommend retrieving the top five to eight chunks and setting a cosine similarity threshold of roughly 0.75. If the highest score is below that threshold, the bot should say the information is unavailable rather than guessing. This single line of logic, which is amply documented across vector database vendor tutorials, is what transforms a chatty random-text generator into a genuinely useful site assistant that lives inside your CRM, your wiki, and your FAQ. RAG will never be a perfect substitute for fine-tuning on deep brand voice, but it is the only approach that offers instant updates whenever you edit a single paragraph in your help center.
Deploy the Widget and Choose an Embedding Strategy
After you have the model and the retrieval layer wired together, the final 10% of the work is presenting the bot on your website. The fastest embed is a small custom chat bubble written with a standard HTML <script>
Get the AI Edge, Weekly
The tools, tutorials, and trends that actually pay — no hype.
