How to Build an AI-Powered Customer Support Chatbot with a Custom Knowledge Base

Your customers expect instant, accurate answers to product-specific questions — “How do I reset my API key?” or “Does your enterprise plan support SAML SSO?” — but your support team can’t be online 24/7, and generic LLM chatbots hallucinate answers because they don’t know your product. Meanwhile, building a custom chatbot that actually understands your documentation feels like a project that requires a dedicated ML team.

Why Most Support Chatbots Fail at Product Questions#

The reason generic chatbot solutions fall short isn’t the language model — modern LLMs are more than capable of generating human-quality responses. The failure is in the knowledge layer. A standard chatbot integrated via an API has no access to your product’s specific documentation, release notes, pricing tiers, or known issues. When a customer asks “Does your Pro plan include custom webhooks?” the chatbot either guesses (and gets it wrong) or responds with a vague deflection like “Please check our website for pricing details.”

This creates a worse experience than having no chatbot at all. Customers who receive incorrect information lose trust, and support tickets generated from chatbot hallucinations actually increase overall support load because the agent now has to correct the misinformation first before addressing the original question.

Retrieval-Augmented Generation (RAG) solves this by connecting the LLM to a searchable index of your actual documentation. Instead of generating answers from the model’s training data — which might not include your product at all — a RAG pipeline retrieves relevant chunks from your support articles, API docs, and knowledge base, then feeds them to the LLM as context. The model answers based on what it retrieved, not on what it remembers.

This guide walks through building that pipeline using open-source tools that run on any cloud VM or local machine — no ML team required.

Step-by-Step: Build Your RAG-Powered Support Bot#

We’ll build a chat endpoint that accepts a user question, searches a vector database of your documentation for relevant passages, constructs a prompt with those passages as context, and streams the LLM’s answer back to the user. The whole stack uses four open-source tools: Qdrant (vector database), OpenAI or Anthropic embeddings, a frontier LLM for generation, and a thin Python FastAPI server.

Step 1: Prepare Your Documentation#

Before you can retrieve relevant content, you need your documentation in a machine-readable format. Start by collecting all the pages you want the chatbot to know about:

• Product documentation pages (guides, tutorials, reference)
• API reference docs (OpenAPI specs, SDK readmes)
• FAQs and knowledge base articles
• Changelog and release notes
• Pricing and feature-comparison pages

Export these as plain text or markdown files, one file per page. If your docs are behind a CMS or knowledge base platform, most export to HTML or markdown. Aim for at least 20–50 pages as a starting point — a few high-quality articles about the most common support questions are more valuable than scraping every page on your site.

Once you have the files, split them into chunks of roughly 500–1000 characters each, with 100-character overlap between chunks. The overlap ensures that when a question matches content near a chunk boundary, the relevant passage isn’t lost. Here’s a Python function that handles this:

from typing import List
import tiktoken

def chunk_document(text: str, chunk_size: int = 800, overlap: int = 100) -> List[str]:
    encoder = tiktoken.get_encoding("cl100k_base")
    tokens = encoder.encode(text)
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunk_text = encoder.decode(chunk_tokens)
        chunks.append(chunk_text)
        start += chunk_size - overlap
    return chunks

Step 2: Generate Embeddings and Index Them#

Each chunk needs to be converted into a vector embedding — a numerical representation that captures the chunk’s semantic meaning. When a user asks a question, we convert that question into an embedding too, then find the chunks whose embeddings are most similar (closest in vector space).

Use any embeddings API that supports a text-embedding-3-small or equivalent model:

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def get_embedding(text: str) -> List[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

Now set up Qdrant, an open-source vector database. The simplest setup is running it in Docker:

docker run -p 6333:6333 -p 6334:6334 \
  -v $(pwd)/qdrant_storage:/qdrant/storage:z \
  qdrant/qdrant

Then create a collection and upload your chunk embeddings:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient("localhost", port=6333)

# Create collection
client.create_collection(
    collection_name="support_docs",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

# Upload chunks with embeddings
points = [
    {
        "id": i,
        "vector": get_embedding(chunk),
        "payload": {"text": chunk, "source": filename}
    }
    for i, (chunk, filename) in enumerate(chunk_with_sources)
]

client.upsert(
    collection_name="support_docs",
    points=points
)

Step 3: Build the Query Pipeline#

When a user asks a question, the pipeline does three things:

1. Embed the question — convert it to a vector using the same embedding model
2. Search the vector database — find the top 3–5 most relevant chunks
3. Generate the answer — feed the retrieved chunks as context to the LLM

from fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI
from qdrant_client import QdrantClient

app = FastAPI()
llm = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
qdrant = QdrantClient("localhost", port=6333)

class Query(BaseModel):
    question: str

@app.post("/chat")
async def chat(query: Query):
    # Step 1: Embed the question
    question_embedding = get_embedding(query.question)
    
    # Step 2: Search for relevant chunks
    search_result = qdrant.search(
        collection_name="support_docs",
        query_vector=question_embedding,
        limit=5
    )
    
    context = "\n\n".join([
        hit.payload["text"] for hit in search_result
    ])
    
    # Step 3: Generate answer with context
    system_prompt = f"""You are a customer support assistant. Answer the user's 
    question based ONLY on the documentation context below. If the context doesn't 
    contain enough information to answer, say so honestly — do not make up answers.
    
    Documentation context:
    {context}"""
    
    response = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": query.question}
        ],
        temperature=0.3,
        stream=True
    )
    
    # Stream the response back
    for chunk in response:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

This endpoint streams the answer token by token, giving users a real-time experience rather than waiting for the full response.

Step 4: Add Source Citations#

One of the biggest trust issues with AI chatbots is that users can’t verify where the answer came from. Add source citations by returning the document source alongside each response:

sources = list(set([
    hit.payload["source"] for hit in search_result
]))

response_with_sources = {
    "sources": sources,
    "stream": generate_stream(query.question, context)
}

Display the sources as clickable links in your chat UI — users can click through to verify the information themselves, which dramatically increases trust in the responses.

Step 5: Handle Edge Cases#

Three edge cases will surface in production:

Out-of-scope questions. When a user asks about something outside your documentation (“What’s the weather in Tokyo?”), the system prompt instructs the LLM to say “I don’t have information about that.” The context will contain no relevant passages, so the model has nothing to base an answer on.

Documentation updates. Your docs change — new features launch, old features deprecate, pricing changes. Set up a weekly re-indexing job that re-chunks and re-embeds your documentation source files. Automate it with a cron job or GitHub Action that runs python reindex.py every Sunday.

Content quality. If your documentation is incomplete, outdated, or poorly written, your chatbot will reflect that. The RAG pipeline is a mirror of your knowledge base — improving the source documents improves the chatbot responses. This is a feature, not a bug: it creates a direct incentive to keep your docs current.

Regular Chatbot vs RAG-Powered Chatbot vs Human Support#

The RAG-powered bot sits in the sweet spot: it handles the 70–80% of support questions that are repetitive and documented, while routing the complex edge cases to human agents. This is the model used by companies like Vercel, Notion, and Linear — their AI chatbots answer common questions instantly, and escalate when the context isn’t sufficient.

Common Mistakes and Solutions#

Mistake: Using the LLM’s training data directly. If you prompt “Answer based on your knowledge” without providing context, the model will use its training data — which might describe a competitor’s product or fabricate features you don’t offer. Always provide retrieved context in the system prompt and instruct the model to stay within it.

Mistake: Chunking too aggressively. Chunks smaller than 200 characters often lack enough context for the LLM to understand what they’re about. Chunks larger than 1500 characters dilute the relevance signal — a chunk that’s 80% about authentication and 20% about billing will rank well for a billing question but contain mostly irrelevant text. The 500–1000 character sweet spot balances precision with context richness.

Mistake: Not tracking what users ask. Every question your chatbot can’t answer is a gap in your documentation. Log out-of-scope questions and review them weekly. Each one represents either a documentation blind spot or a feature request from your customers.

Mistake: Skipping evaluation. Before you launch, create a test set of 30–50 question-answer pairs from your actual support tickets. Run them through your pipeline and manually check the answers. Measure how many are correct, how many cite the right source, and how many are wrong. Iterate until you hit at least 85% accuracy.

Build Your First Support Bot This Week#

A RAG-powered customer support chatbot is one of the highest-ROI AI projects you can build as a solo developer or small team. The setup takes a weekend: one day to prepare and index your documentation, one day to build the API endpoint and integrate it with your chat widget (Intercom, Crisp, or a simple embedded iframe).

The tools are all open-source and well-documented. Qdrant’s free tier handles up to 1 million vectors — enough for thousands of documentation pages. The embeddings API costs pennies for most documentation sets. The LLM inference costs a fraction of a cent per query.

For a deeper look at how AI agents can automate more of your workflow, read our guide on how to build an AI research pipeline. If you’re evaluating vector database options for your RAG stack, see the Qdrant documentation — it has Python, TypeScript, and Go client libraries with the same search interface.