Building a chatbot on your own data (RAG): architecture, cost and pitfalls

A retrieval-augmented generation (RAG) chatbot answers questions from your own documents instead of guessing from what a general-purpose model learned in training. Good RAG chatbot development pairs a search system over your content with a language model that is told to answer only from what it retrieves and to show where each answer came from, and keeping it current means updating an index rather than retraining a model. This guide covers the architecture end to end, what drives the running cost, and where these systems usually go wrong.

If you're still deciding whether an AI project is worth doing at all, start with our guide to custom software development or practical AI automation for SMEs.

What "RAG" actually means#

A general-purpose language model only knows its training data plus whatever you put in the prompt. RAG fixes the "it doesn't know our stuff" problem without retraining anything: when someone asks a question, the system searches your documents for the most relevant passages, inserts them into the prompt, and asks the model to answer using only that material. The model does the writing; your documents supply the facts. That is also what makes citations possible, because the system knows exactly which passages it handed over.

The term comes from a 2020 research paper by Patrick Lewis and colleagues, which paired a pre-trained language model with a neural retriever over a dense vector index of Wikipedia (Lewis et al., 2020). AWS's plain-English definition: RAG is "the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response" (AWS).

The pipeline, step by step#

A production RAG system has nine parts:

1. Ingestion. Pull documents from where they already live (wiki, shared drive, CMS, PDFs, helpdesk history), convert them to plain text, and keep a reference back to the original file, page or URL.
2. Chunking. Split each document into passages small enough to embed and retrieve individually, typically a few hundred tokens with some overlap so a fact isn't cut in half. For structured documents such as policies or manuals, splitting on headings and paragraphs usually retrieves better than fixed-size chunks.
3. Embeddings. Convert each chunk into a numeric vector with an embedding model, so passages with similar meaning sit close together. The same model must embed the user's question at query time.
4. Vector store. Index those vectors for fast nearest-neighbour search as the collection grows from hundreds of chunks to millions.
5. Retrieval. Embed the question and fetch the nearest chunks. Many systems add keyword search as well ("hybrid retrieval"), because embeddings can blur exact terms such as part numbers, error codes or names.
6. Reranking. An optional second pass in which a more expensive model re-scores the top candidates, trading some latency and cost for better precision.
7. Prompting. Build a prompt from the retrieved chunks plus an instruction to answer only from that material and to say plainly when the answer isn't there.
8. Citations. Return the chunks behind each part of the answer, so a person can check it against the source. A simple response contract might look like this:

{
  "answer": "Refunds are processed within 5 business days of approval.",
  "citations": [
    { "source": "refund-policy.pdf", "chunk_id": "refund-policy.pdf#12", "score": 0.87 }
  ]
}

9. Evaluation. Keep measuring whether it works: retrieval recall against a known-answer test set, whether answers are faithful to the retrieved text, and regular human review of real conversations.

Choosing components for RAG chatbot development#

None of these layers locks you into one vendor. Here are the six main decisions, with real options and a rule of thumb for each:

pgvector is worth singling out: it's an open-source extension that adds vector storage and nearest-neighbour search, including HNSW and IVFFlat indexes, directly to PostgreSQL (pgvector). If you already run Postgres, it saves standing up and paying for a separate vector database.

What actually drives the cost#

There is no single "RAG chatbot price". Cost comes from one-off ingestion, storage that grows with the number of chunks, query costs that grow with usage, and hosting for anything you run yourself. You can estimate each before you build.

• Ingestion (embedding) cost is the number of chunks × tokens per chunk × the embedding price per token. To estimate tokens, OpenAI's embeddings guide works on roughly 800 tokens per page (OpenAI); for an exact count, run your own documents through the provider's tokenizer. As at September 2026, OpenAI listed text-embedding-3-small at US$0.02 and text-embedding-3-large at US$0.13 per million tokens (OpenAI pricing). Worked example: 5,000 chunks of about 300 tokens is 1.5 million tokens, or roughly US$0.03 with the small model. For most SME document sets, ingestion is not the big cost.
• Storage is the number of chunks × embedding dimensions × 4 bytes (for 32-bit floats), plus index overhead. text-embedding-3-small produces 1,536 dimensions by default and -large 3,072 (OpenAI), so those 5,000 chunks take about 31 MB of raw vectors with the small model. pgvector on a server you already run adds no separate storage bill. Managed stores charge per GB plus read and write operations: as at September 2026, Pinecone listed US$0.33 per GB per month on its Standard and Enterprise plans, per-million read and write unit charges, and a free Starter tier with capped usage (Pinecone pricing).
• Query cost grows with usage, not document volume. Retrieval itself is cheap. Reranking adds a per-query cost: Cohere measures rerank usage in "search units" of one query against up to 100 documents (Cohere pricing; Rerank overview). Generation is priced per input and output token: as at September 2026, OpenAI listed gpt-4o-mini at US$0.15 per million input tokens and US$0.60 per million output tokens. Retrieved chunks count as input tokens on every question, so five 300-token chunks add about 1,500 input tokens to each answer.
• Hosting covers wherever the vector store and any self-hosted models run. A self-hosted pgvector index needs a server sized for Postgres, with the index held largely in memory; our managed application hosting takes care of sizing and patching if you'd rather not.

Treat every figure here as dated and illustrative. The honest way to budget is to estimate your own chunk and token counts against the current pricing pages, then check the estimate with a short pilot on real questions.

Privacy and data residency#

Where your data goes depends on the components you choose. With hosted model APIs for embeddings or generation, your document text and users' questions leave your infrastructure and are processed on that provider's servers. With fully self-hosted components (an open-weight embedding model, an open-weight language model and a self-hosted vector store), nothing needs to leave a server you control.

Residency is only half the question. A self-hosted vector store doesn't help if the embedding or generation step still calls a third-party API: that request, including the retrieved passages and the user's question, is handled under the vendor's own data-handling and retention terms. Read the current API and enterprise terms of any model provider before deciding what you're willing to send it.

This matters most when data must stay in a particular country or region. Inventure resells and manages OVHcloud infrastructure in Sydney, Singapore, Mumbai, Frankfurt, London and Canada, so clients in Australia and Nepal can choose where a self-hosted RAG stack physically runs. This is general information about architecture options, not legal advice about what a particular privacy law requires of your organisation.

Failure modes, and how to test for each one#

RAG systems rarely fail loudly. They fail by sounding confident while being wrong, so deliberate testing matters more than usual:

OWASP's Top 10 for LLM Applications lists prompt injection as LLM01 and covers retrieval-specific risks under LLM08, Vector and Embedding Weaknesses. Position matters too: research on long contexts found that performance is often highest when the relevant information sits at the beginning or end of the input, and degrades when it sits in the middle (Liu et al., 2023), so test with the correct chunk in different positions. Whatever you test, rerun the same evaluation set as a regression suite before changing the prompt, the model or the embedding model.

Do you need to fine-tune anything?#

Usually not. Fine-tuning changes how a model behaves: its tone, its output format, its handling of a narrow and stable task, or its consistent use of domain jargon. It's a poor way to teach facts that change, and it doesn't produce citations. Nor will fine-tuning alone reliably stop a model inventing facts it never saw in training, because it changes behaviour rather than what the model knows. RAG suits knowledge that is large, sensitive or often updated, which describes most internal documentation, policies and product catalogues.

The two can work together. Some teams fine-tune a model to follow "answer only from the provided context, and say when you don't know" more reliably, while RAG still supplies the facts. Start with RAG alone, measure where it falls short, and consider fine-tuning only to fix a specific, measured gap.

What to do next#

A working RAG assistant is mostly an integration and evaluation project: connect a document source, a vector store and a model, then prove it answers correctly before anyone relies on it. Our AI automation team can scope the pipeline and the hosting together, including whether to self-host for data residency. If what you need is closer to processing documents end to end, see our guide to turning a web form into a signed PDF.