Chat widgets have been on websites for years, but most early ones were frustrating decision trees that understood nothing outside their script. Large language models have changed what is possible: a modern AI chatbot for website visitors can understand questions phrased in many ways and answer in natural language. That flexibility is also the risk. A bot that can say anything can say the wrong thing, confidently, in your company's name. This article covers the decisions to make before you add one.
Three kinds of website chatbot
| Type | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Rule-based | Buttons and scripted flows | Fully predictable, cheap | Rigid; fails on anything unexpected |
| General LLM | A language model answers from its general training | Fluent, flexible | Knows nothing specific about your business; may invent answers |
| LLM with retrieval | Searches your approved content, then the model answers from what it found | Flexible and grounded in your information | Only as good as your content; needs setup and testing |
For most businesses, the third type is the right choice. The technique behind it is called retrieval-augmented generation (RAG): the system retrieves relevant passages from your documents and instructs the model to answer only from them. Some bots combine approaches, using fixed flows for sensitive tasks such as cancellations and the language model for open questions.
Decision 1: What is the bot for?
"Answer customer questions" is too broad. Define a narrow, testable scope, for example:
- Answer questions about products, pricing pages, shipping and returns policy.
- Help visitors find the right service page or document.
- Collect details for a quote request and pass them to sales.
- Check order status for logged-in customers via your order system.
Equally, write down what it must not do: give legal, medical or financial advice, negotiate prices, promise delivery dates it cannot verify, or discuss competitors. These boundaries go into the bot's instructions and into your test cases.
Decision 2: What will it know?
The bot's answers can only be as good as the content behind them. Before launch, gather and tidy the sources: FAQs, policies, product information, help articles. Outdated or contradictory pages will produce outdated or contradictory answers. Decide who owns each source and how updates flow into the bot, ideally automatically when a page is published.
Ask the bot to cite or link the page each answer came from. Visitors can check, and your team can trace bad answers back to their source.
Decision 3: How does it hand over to a human?
Nothing frustrates customers more than being trapped with a bot. Plan the handoff:
- Offer a visible "talk to a person" option at all times.
- Escalate automatically when the bot is unsure, when the visitor is clearly upset, or after repeated failed attempts.
- Pass the conversation transcript to the human so the customer does not have to repeat themselves.
- Outside office hours, collect contact details and set an honest expectation of when someone will reply.
Decision 4: Privacy and data handling
Visitors type all kinds of things into chat boxes, including phone numbers, addresses and complaints with personal details. Consider:
- Where conversations go. If you use an external AI provider, check its terms on data retention and whether your data is used to train models. Business and API plans often differ from consumer ones.
- What you store. Keep transcripts only as long as needed, restrict who can read them, and include the chatbot in your privacy notice.
- Identity. If the bot looks up order or account information, it must verify who it is talking to through your normal login, not by trusting what the visitor types.
- Disclosure. Tell visitors they are talking to an automated assistant. Rules on this vary by country, and it is good practice regardless.
Decision 5: Security
A chatbot is a public text box connected to a powerful system, and people will try to misuse it. The main risk is prompt injection: messages crafted to override the bot's instructions ("ignore your rules and..."). Practical defences:
- Give the bot the minimum access it needs. A bot that only reads public help content cannot leak private data.
- If it calls internal systems, use narrow, read-only functions with permission checks on your server, never broad database access.
- Never put secrets, internal pricing logic or confidential notes into its instructions or knowledge base.
- Rate-limit usage to control abuse and cost.
- Review the OWASP Top 10 for LLM Applications for the wider list of risks.
Decision 6: Cost
LLM services usually charge per token, a unit of text roughly corresponding to a word fragment, for both the question and the answer, including the retrieved content sent along with each question. Costs scale with traffic and conversation length. Estimate using realistic volumes, set spending limits, and monitor usage. Add the platform or development cost, hosting, and staff time to maintain content and review conversations.
Testing before launch
Build a test set of real questions from your support inbox, plus awkward ones: off-topic requests, attempts to get discounts, questions your content does not answer, abusive messages and injection attempts. Check that the bot answers correctly, admits when it does not know, stays within scope and hands over properly. Re-run the same tests whenever you change the model, instructions or content.
After launch: review and improve
Read a sample of conversations every week. Track how often the bot resolves questions without escalation, how often visitors ask for a human, and thumbs-up or thumbs-down ratings. Unanswered questions are valuable: they show gaps in your website content as well as the bot.
When not to add an AI chatbot for website visitors
If your site gets few visitors, your questions are mostly complex and bespoke, or you do not have staff to maintain content and handle escalations, a clear FAQ page and a good contact form may serve customers better. Our AI and machine learning development team builds grounded chatbots, and our web application development team integrates them with order and account systems.
Key takeaways
- Ground an AI chatbot for your website in your own approved content, with links to sources.
- Define a narrow scope and explicit no-go topics.
- Always offer a smooth handoff to a person.
- Plan for privacy, prompt injection and usage costs before launch, and review conversations afterwards.