You don't train a chatbot on your business documents. You give it those documents to read each time a customer asks something. That approach is called retrieval-augmented generation, or RAG, and for a small company it often isn't needed at all: if your material is under roughly 500 pages, Anthropic says you can paste all of it into the prompt.
How do you train an AI chatbot on your own business documents?
You don't retrain the model. You connect it to your documents so it looks up the relevant passages before it answers. OpenAI's documentation describes this as giving the model access to domain-specific context. The model's own knowledge stays as it was; the answer is built from your text.
"Training" is the word vendors use because it sounds simple. Underneath, a typical setup splits your files into chunks, stores them in a searchable index, finds the few chunks closest to the customer's question and hands them to the model with an instruction to answer from them. The idea comes from a 2020 research paper by Patrick Lewis and colleagues, which paired a language model with a searchable index of Wikipedia.
Do you need RAG at all? The 500-page test
Often not. Anthropic's guidance on retrieval says that if your knowledge base is smaller than 200,000 tokens, about 500 pages of material, you can include the whole thing in the prompt, with no need for RAG. It adds that prompt caching makes this faster and cheaper, with latency down more than two times and costs down by up to 90 percent.
Count what you actually have. A price list, a returns policy, thirty service descriptions, an FAQ and your last two years of support answers fit in that limit for most small businesses. The vendor guides that dominate search results for this question jump straight to vector databases. For a business with a few dozen pages, that's building a warehouse to store a shoebox.
Two caveats. The 200,000 figure is Anthropic's, for its own models, so check the context window of whichever model you use. And a long prompt costs money on every question, so it suits a modest volume of chats. Past that size, retrieval starts to earn its complexity.
Why chatbots answer wrong even with your documents
The usual culprit is the chunking step. Anthropic's 2024 write-up gives an example: a chunk that says revenue grew 3 percent doesn't say which company or which quarter, so a question about that company's results may never find it. The same thing happens on a business site. A chunk reading "Delivery takes 3 to 5 working days" doesn't say which product line or country it applies to.
Anthropic measured the effect. With standard retrieval, 5.7 percent of questions failed to surface the right passage in the top 20 results. Adding a line of context to each chunk cut that to 3.7 percent. Combining it with keyword search cut it to 2.9, and adding reranking brought it to 1.9. That's a 67 percent reduction in failures, from a change to how the documents are prepared rather than a bigger model.
You can borrow the lesson without any of the tooling. Write your source documents so that every paragraph survives being read alone.
Rewrite for the paragraph that gets cut loose
Name the product, the region and the date inside the sentence that depends on them. Replace "it", "this plan" and "the above" with the thing itself. Put one question and one answer under each heading rather than a long page that mixes five topics. It's dull editing work, and it fixes the problem at its source.
Strip what the bot shouldn't repeat
Navigation menus, cookie banners and footers are text too. If a tool crawls your site as-is, they go into the index with everything else. Feed it the content, not the page chrome.
RAG or fine-tuning: which does a business need?
Almost always RAG. OpenAI's guide separates the two problems: RAG is for when the model lacks knowledge, because your prices and policies weren't in its training data. Fine-tuning is for when the model's behaviour is inconsistent, such as formatting or tone. The guide calls the two methods additive, not exclusive, but the first problem is the one most businesses have.
There's a practical reason too. The Lewis paper lists updating a model's knowledge as an open problem for models that store facts in their parameters. With retrieval, you edit a document and the next answer changes. With fine-tuning, a price change means another training run.
What to test before you put it on your site
Write twenty real questions from your inbox. Include five whose answer is nowhere in your documents. A good setup says it doesn't know and points to a person; a bad one invents a policy. Check whether each answer can be traced to a passage, because the Lewis paper names provenance, knowing where an answer came from, as one of the hard open problems.
If you want an assistant built around your own content, the AI integration and automation side of what onlyspell does covers exactly this: deciding whether you need retrieval, preparing the material and testing the answers. To discuss a specific case, talk to us about your project.
Frequently asked questions
Can I train ChatGPT on my own files?
Not in the sense of changing the model. You give it your files as reference material, through a retrieval system or, for a small set, directly in the prompt. The model reads the relevant passages when it answers, and its own training stays as it was.
How many documents can a chatbot handle without a vector database?
Anthropic says a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt. Above that, you need retrieval to pick the relevant parts. The limit depends on the model you use, so check its context window.
Why does my chatbot make up answers?
Usually the right passage wasn't retrieved, or it was retrieved without enough context to be used. Anthropic found that adding context to each chunk cut failed retrievals from 5.7 percent to 1.9 percent when combined with other methods. Rewriting documents so each paragraph stands alone helps, and so does telling the bot to say when it doesn't know.
Sources
If you still need additional help in developing and designing your website, our magic is here to help. Contact us today, and our wizards will help you immediately with your digital marketing needs!
