First, save the exact wrong answer
An AI chatbot giving wrong answers does not automatically need a new model. The failure may sit in the website content it reads, the retrieval step that selects that content, the instructions that shape the reply, or an integration that cannot reach current business data.
Do not begin by rewriting one large prompt until the bot sounds better. Save the customer’s exact question, the complete answer, the time, the page where the chat started, and any sources or links the bot displayed. Remove names, email addresses, order numbers, and other private data before sharing the record.
Then ask the same question in a new conversation. If the answer changes, keep both versions. That variation is useful evidence because it separates a consistently stale source from an answer path that changes between requests.
AI chatbot giving wrong answers: find the failing layer
| What the customer sees | What it usually narrows down | First safe check |
|---|---|---|
| The same old price, policy, or opening time appears every time | Outdated source page, duplicate document, or an index that was not refreshed | Find the exact source text and confirm its owner, URL, update date, and indexed version |
| The correct answer exists on the website, but the bot ignores it | Retrieval, page access, content chunking, source priority, or a question that does not match the stored wording | Ask for the source and inspect what the bot retrieved before changing the answer prompt |
| The bot invents a discount, guarantee, delivery time, or service promise | Missing answer boundary, weak fallback, or an instruction to be helpful even when evidence is absent | Test whether it can say that the information is unavailable and offer a human handoff |
| General questions work, but order, booking, stock, or account answers fail | Live-data integration, authentication, permissions, API mapping, or stale cached data | Confirm which system owns the current value and whether the bot actually has read access to it |
| The answer is useful, but its page link is missing or redirects elsewhere | Old URL in the knowledge source, retired content, redirect handling, or generated links that were never verified | Open every cited link anonymously and compare it with the site’s canonical URL |
| A specially phrased question makes the bot ignore its role or reveal unrelated information | Prompt injection, excessive permissions, untrusted retrieved content, or missing output controls | Stop public testing and review the bot’s trust boundaries, tools, and data access |
This matrix is a starting point, not proof of one cause. A stale policy can coexist with weak retrieval, and a correct knowledge source does not protect a bot that has broader permissions than it needs.
Run a ten-question truth test
Test questions should represent customer work, not a demo script written to make the bot succeed. Create a small set that you can repeat after every content, model, plugin, or integration change.
- Start with a simple question whose answer appears clearly on one current page.
- Rephrase it using the words a new customer would use.
- Check a price, deadline, policy, or availability that changed recently.
- Use a question whose answer appears on two pages and check which source wins.
- Request a link to the relevant page and open it in a private browser window.
- Cover a question the website does not answer.
- Include an ambiguous question that needs a follow-up before a safe answer is possible.
- Request account, booking, order, or stock information without signing in.
- Try to reach a person during and outside support hours.
- Repeat one important question in a new conversation and compare the wording, facts, and source.
Record four outcomes for each question: the expected answer, the actual answer, the source used, and the correct fallback. A polished reply still fails if it states the wrong policy or sends the customer to a dead page.
Separate source, retrieval, and behavior
OpenAI’s current knowledge retrieval guidance describes a three-part path: ingest the source data, configure retrieval, then run evaluations before deployment. That order is useful even when your chatbot uses another platform.
Source means the pages, documents, product records, policies, and other facts the bot may use. Retrieval is the step that chooses relevant pieces for a question. Behavior covers the instructions, answer format, uncertainty rule, citations, tools, and escalation path.
Fix the layer that failed. A prompt cannot make an outdated refund policy current. Re-uploading documents will not repair an API permission error. Replacing the model will not fix two website pages that disagree about the same service.
Give every important fact one owner
Start with facts that could lose money or trust: prices, service scope, delivery dates, returns, cancellations, guarantees, opening hours, stock, and contact routes. For each one, record a canonical source and the person responsible for keeping it current.
Remove or redirect duplicate pages only through the site’s normal content and SEO process. Do not delete a document from the bot first if customers or staff still depend on it elsewhere. Instead, mark the authoritative source, resolve the contradiction, then refresh the bot’s index and retest the affected questions.
Add visible dates where timing matters. A policy page with no effective date can look current to a reader while an old copy remains in the chatbot’s knowledge store. Version names also help when the answer changes by plan, product, country, or contract.
Check what the bot retrieved
If the platform can show sources, citations, retrieved passages, or request logs, inspect them for the failed question. You are looking for three different states:
- the correct source was never available to the chatbot;
- the source was available, but retrieval selected the wrong passage;
- the correct passage was supplied, but the final answer changed or extended its meaning.
Missing source access needs a content or indexing repair. Wrong passage selection may need better page structure, document splitting, metadata, source priority, or query matching. Unsupported additions need tighter answer rules, output checks, or a different workflow for that question.
Do not feed the bot an entire unstructured export and assume more data means better answers. Remove navigation noise, expired offers, repeated footers, test pages, draft policies, and private material that does not belong in customer support.
Make “I do not know” a valid result
OpenAI’s current accuracy guidance notes that language models can produce confident but incorrect information. A customer-facing bot therefore needs a deliberate answer for missing or uncertain evidence.
Define what the bot may answer, what requires a follow-up question, and what must go to a person. A useful fallback can say that the available information does not confirm the answer, link to the relevant contact route, and pass the question plus approved conversation context to support.
Avoid promising that every answer is accurate. Show the source when practical, especially for policies and instructions, and make the human route easy to find without forcing the customer to argue with the bot.
Do not confuse public information with live customer data
A knowledge chatbot may answer from public pages without access to orders or accounts. If it claims that a particular parcel shipped, a booking moved, or a product is in stock, it needs a current system connection with the correct permissions and identity checks.
Use the least access needed. A support bot that only reads order status should not be able to refund, cancel, edit customer details, or send messages unless the business has designed and approved those actions separately.
When a live action carries financial, privacy, legal, or account risk, require deterministic validation and human approval where appropriate. Do not rely on a persuasive natural-language answer as proof that an action succeeded.
Test the security boundary separately
A wrong answer can be ordinary misinformation, but it can also follow an attempt to change the bot’s instructions. OWASP’s prompt injection guidance recommends constrained behavior, validated outputs, least-privilege access, separated untrusted content, human approval for high-risk actions, and adversarial testing.
Do not publish attack strings or test a live customer bot with real private data. Use an isolated test environment and harmless questions that verify the boundary: can a user make the bot leave its support topic, cite unapproved content, expose hidden instructions, or call a tool it should not use?
If the bot can reach private records or perform actions, stop public experimentation when a boundary fails. Preserve the conversation and request logs, revoke unnecessary access, and ask the integration owner to review the complete tool path.
Use this repair boundary
| Evidence | Safe owner action | When specialist help is appropriate |
|---|---|---|
| One public page contains an old fact | Correct the approved source, record the change, refresh the index, and rerun the truth test | Several systems copy the fact or nobody owns the canonical version |
| The bot cannot find a current public answer | Confirm that the page is published, readable, and included in the configured source set | Retrieval, document parsing, permissions, or custom indexing needs repair |
| The correct passage is retrieved but the answer adds unsupported details | Strengthen the fallback and require sources for that answer class | You need evaluation rules, output validation, or workflow redesign |
| Order, stock, booking, or account data is wrong | Stop presenting it as live and verify the system of record | The API, authentication, cache, field mapping, or permissions are incorrect |
| A user can override boundaries or trigger an unapproved action | Disable the affected action and preserve the evidence | A security and integration review is required before re-enabling it |
Retest after every meaningful change
Run the same ten questions after a source refresh, prompt change, model update, plugin update, or API change. Compare facts, links, source selection, refusal behavior, and human handoff. Keep a small release record so a future failure can be tied to the change that introduced it.
Test on desktop and mobile, signed in and signed out where relevant. Confirm that the widget loads, consent behavior is correct, keyboard access works, messages fit the screen, links open safely, and a failed service returns a useful fallback instead of an endless spinner.
Webless WordPress development services can repair a chatbot’s content connection, retrieval path, API mapping, permissions, and handoff workflow. Ongoing WordPress maintenance can add repeatable tests around site, plugin, content, and integration changes. For a useful first review, send the expected answer, one redacted wrong answer, the source page, and the affected integration through the Webless contact page.
Fix the answer path, not only the wording
A website chatbot can sound confident while using an old source, retrieving the wrong passage, crossing a permission boundary, or guessing when it should hand the question to a person. Replacing the model may leave every one of those problems intact.
Start with one saved failure and a ten-question truth test. Verify the canonical source, inspect retrieval, define safe answer and action boundaries, then retest the same customer questions. That gives you a repair you can explain and monitor instead of a prompt change that only makes the next demo look better.