AI Chatbot Giving Wrong Answers? How to Test It Before Customers Do

When an AI chatbot giving wrong answers turns up on a business website, the cause is usually one of two things: it has no current, approved source for the fact it was asked about, or nothing tells it to stop and say so when it does not know. You can find many of those answers before a customer does. Write down the facts customers ask about, ask the chatbot about each one the way real people ask, compare every reply to your own records, and mark it pass or fail.

This guide is that test, written for an owner who has a website chatbot or is about to add one. It covers what to ask, how to spot an invented or outdated answer, what the bot should do when it has no answer, and when a person needs to take over. If you are still deciding what kind of tool you need, start with AI chatbot vs AI website assistant and come back here before launch.

Why Is Your AI Chatbot Giving Wrong Answers?

The language model behind an AI chatbot is built to produce a plausible reply. OpenAI's explanation of why language models hallucinate defines hallucinations as "plausible but false statements generated by language models," and argues that standard training and evaluation "reward guessing over acknowledging uncertainty."

That is a general description of how these models behave, not a description of any one product. On a business website, it shows up in a few ordinary ways:

• No source for the fact. The bot was never given your holiday hours, so it offers the hours that sound normal for a business like yours.
• A stale source. It was given last spring's price sheet, and it repeats it with full confidence.
• Two sources that disagree. The website says one thing, an uploaded PDF says another, and the bot picks one.
• No permission to stop. Its instructions push it to be helpful, and nothing says that "I don't have that information" is an acceptable reply.

The first three are problems with the facts. The fourth is a problem with the rules. A test needs to catch both.

Give It One Set of Current Facts First

Before testing anything, write down the facts the chatbot is allowed to use, in one place, with someone responsible for keeping them current. That document is what you will grade every answer against.

• Hours, including holidays, seasonal changes, and any difference between phone, walk-in, and delivery hours.
• Services or products you offer, and the ones you do not.
• Prices, or how pricing works. When price is part of the customer's decision, the bot needs real numbers or a clear rule: a fixed price, a starting price, a minimum, or "priced after an on-site visit." A vague answer about price either loses the customer or sets an expectation the staff then has to undo.
• Availability: what can be booked, ordered, or scheduled, and how far ahead.
• Policies: cancellations, deposits, refunds, warranties.
• Service area: the towns, ZIP codes, or delivery radius you actually serve.
• Contact details: phone, email, address, and which one to use for what.
• The next step: how a customer books, orders, or sends an inquiry, in the order it really happens.

These are the same facts search engines and AI answer systems look for. Google's guidance on improving local ranking says "businesses with complete and accurate info are more likely to show up in local search results," and lists hours among the details to provide. When your chatbot, your website, and your Business Profile carry the same hours, prices, and service area, a customer gets one answer wherever they ask. When they disagree, the customer gets two answers and no way to know which is right. How those three kinds of search fit together is covered in SEO vs AEO vs GEO.

What to Ask: Four Kinds of Questions

Customers do not type the questions you wrote in your FAQ. Test each fact with four kinds of question.

• Normal. "What time do you close on Saturday?" The bot should give the right answer, briefly.
• Unclear. "are u open late," "how much for a big one," "can you come out tmrw." The bot should ask a short clarifying question or answer the likely meaning and say what it assumed.
• Another language. If your customers write in Spanish, test in Spanish, including mixed-language messages. Check that the facts survive translation: prices, hours, and addresses should match exactly.
• Outside what the business knows. "Is your chicken from a local farm?" "Will this fix the leak for good?" "What's the wait right now?" If the answer is not in your facts, the right reply says so.

Ask each question more than once, in a fresh conversation each time. Identical wording can produce a different reply on a later run, and an answer that passes once has passed once.

How to Spot an Invented or Outdated Answer

Read every reply next to your fact sheet and look for four problems.

• Invented. A detail that appears nowhere in your facts: a discount you never offered, a service you do not provide, a policy that sounds reasonable but does not exist. Specific numbers deserve the closest read, since an invented one looks exactly like a real one.
• Stale. A fact that used to be true. Old hours, last year's prices, a closed location, a dish no longer on the menu.
• Missing escalation. The bot should have handed the question to a person and instead answered it, or kept the customer going in circles.
• Overconfident wording. "Guaranteed," "always," "definitely available," "no problem." A promise the business would not make in person is a failed answer even when the underlying fact is correct.

What Should the Chatbot Do When It Doesn't Know?

The correct reply to a question outside its facts has three parts. It says it does not have that information. It does not guess. It offers the next step: the phone number, the inquiry form, or a handoff to a person with the question already written down.

A good example: "I don't have today's wait time. You can call the front counter at the number below, or leave your name and party size and someone will text you." That answer passes. A cheerful estimate of "about fifteen minutes" fails, even if it happens to be right that day.

Some questions should go to a person every time, whatever the bot knows:

• Judgment calls, such as allergies, safety, a diagnosis of a problem, or whether something is covered by a warranty.
• Exceptions, such as refunds, a price outside the published range, or a request to bend a policy.
• A request for a person. If the customer asks for a human, the bot should not argue.
• An upset customer. A complaint is a conversation for staff.

The handoff should carry the conversation with it, so the person who picks it up can see what was asked and the customer does not start over. Routing that handoff into your phone, inbox, or CRM is a system question, and API integrations for small businesses covers how those connections work.

A Simple Pass/Fail Record

Keep the test in a spreadsheet so it can be run again. One row per question, with these columns:

• Fact being tested and where the correct answer comes from
• The exact question asked, including typos and language
• Run number and date
• The bot's reply, copied word for word
• Result: pass or fail
• Failure type: invented, stale, missing escalation, or overconfident
• Fix made and the date it was re-tested

Two rules make the record useful. First, a failure on price, a policy, or anything involving health or safety blocks launch until it is fixed and re-tested. Second, the questions stay saved. Every time the hours, prices, menu, policies, bot instructions, or underlying model change, run the same questions again and compare against the last record. A fix to one answer can quietly change another.

A Fictional Example: A Restaurant Tests Its Chatbot

The restaurant below is made up, and so are its details.

A family restaurant adds a chatbot to its website before the holiday season. The owner writes a one-page fact sheet: hours, the current menu with prices, the catering minimum, a three-mile delivery radius, the reservation policy for groups of eight or more, and the phone number for the front counter. Then two staff members spend an afternoon asking it questions, in English and in Spanish, from their phones.

Most answers pass. Five do not.

• Holiday hours. Asked "open on thanksgiving?", the bot gives the regular Thursday hours. The fact sheet never mentioned holidays. Stale by omission. Fix: add the holiday schedule.
• Catering price. Asked about catering for 40, it quotes a per-person price from an old catering PDF still on the website. Stale. Fix: remove the old PDF and add the current catering minimum and per-person starting price.
• Delivery area. Asked in Spanish whether they deliver to an address five miles away, it says yes. Invented. Fix: state the radius as a rule and tell the bot to offer pickup outside it.
• Allergy question. Asked whether a dish is safe for a severe nut allergy, it says "Yes, it's nut-free." Missing escalation and overconfident. Fix: allergy questions always go to staff, with the counter phone number.
• Large party. Asked to "book a table for 12 friday," it confirms a reservation it has no way to make. Invented. Fix: explain the group policy and send the request to the reservation inbox.

After the fixes, the staff run the same saved questions again in fresh conversations. The five failures now pass, and two earlier passes are checked again because the fact sheet changed around them. The record goes in a shared folder, dated, and the owner re-runs it when the spring menu comes out.

Who Is Responsible for What the Chatbot Says?

The business is. In Moffatt v. Air Canada, a February 2024 decision from British Columbia's Civil Resolution Tribunal, a customer relied on the airline's website chatbot for its bereavement fare policy and got wrong information. Air Canada argued the chatbot was responsible for its own statements. The tribunal called that "a remarkable submission" and wrote: "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot."

That is one small-claims decision in one Canadian province, not a rule for every court. The practical point holds anywhere: a customer reads the chatbot's reply as the business speaking. This article is operational guidance, not legal advice.

How LoGa Builds and Tests Website Assistants

LoGa AI Systems is an Oklahoma City company that builds custom software and business systems for service businesses, including the website assistants that answer customer questions. Our approach follows this guide: one approved set of current business facts, a bot that answers only from them, a clear "I don't know" with a next step, handoffs to a person for judgment calls, and the same saved test run again after every change. No one can promise a chatbot that is never wrong, and we don't. What a test record gives you is knowing which answers have been checked and when.

How the assistant fits with the rest of the system is on the platform page, and AI website tools for small businesses covers which website features are worth paying for. If you want your own customer questions and handoffs reviewed, book a free bottleneck audit.

Frequently asked questions

Why does an AI chatbot make up answers? A language model writes the most plausible reply it can. When it has no current, approved source for a fact, a plausible guess can read exactly like a real answer. The fix is to give it one approved set of business facts, tell it to answer only from those facts, and give it a clear way to admit a gap and pass the question to a person.

How do I test an AI chatbot before launch? Write down the facts customers ask about most: hours, services, prices or how pricing works, availability, policies, service area, contact details, and the booking or inquiry steps. Ask about each one the way customers really ask, including unclear, misspelled, and other-language versions, plus questions the business cannot answer. Compare every reply to your source and mark it pass or fail.

What should a chatbot say when it does not know the answer? It should say plainly that it does not have that information, avoid guessing, and offer the next step: a phone number, a contact form, or a handoff to a person with the question already written down. A short honest reply that leads somewhere is better than a confident one that is wrong.

When should a chatbot hand a conversation to a person? When the question is outside its approved facts, when the answer depends on judgment such as allergies, safety, refunds, or exceptions to a policy, when the customer asks for a person, or when the customer is upset. The handoff should carry the conversation with it so the customer does not have to repeat themselves.

How often should I re-test my chatbot? Every time something it relies on changes: hours, prices, the menu or service list, policies, the instructions it follows, or the model behind it. Run the same saved questions again and compare the results to the last record. A quick spot-check on a regular schedule also catches facts that changed without anyone updating the bot.