After launch: a routine for improving your bot

Your bot will never be as good on launch day as it is a month later. Real customers ask things you didn't expect, word questions differently from your help pages, and find the gaps in your content. A short, regular review turns those conversations into better answers. This guide gives you a routine for the first few weeks, then a lighter weekly one.

What you'll use, by plan

On the Free and Basic plans the routine still works: you just read conversations yourself instead of relying on summaries and analytics.

Before you go live

Two settings make the rest of the routine much easier. Both are in your bot's General settings tab.

• Turn on Allow users to provide feedback on responses. Customers then see thumbs up and down buttons on each answer, and you can filter for thumbs down.
• Turn on Include sources for responses while you're reviewing. Seeing which pages an answer came from is the quickest way to tell a knowledge gap from a retrieval problem. You can turn it off again later if you don't want customers to see links.

If you haven't already, work through Check your bot is ready to launch.

1. Find the conversations worth reading

You don't need to read every chat. Open Conversations, choose your bot, and use Filters to go straight to the ones most likely to show a problem:

• Feedback set to thumbs down: customers telling you an answer was wrong.
• State set to Escalated: the bot handed off, which often means it couldn't help.
• Sentiment set to Negative, or Emotions such as Frustrated or Confused (Pro and Enterprise).
• Search for phrases your bot uses when it's stuck, such as "sorry" or "don't know", and for words like "human", "person" or "agent".

In the first week, look daily. After that, a weekly look is usually enough. The Summary in each conversation's side panel (Standard plan and above) tells you what a chat was about without reading it all; summaries appear about 10 minutes after the last message.

2. Work out why each answer went wrong

For each bad answer, find the cause before changing anything. Otherwise you risk fixing the wrong thing, for example rewriting your instructions when the page simply never synced.

• Ask Otto. Open the conversation and click Ask Otto about this chat. Otto reads the exchange and the sources the bot used, explains the cause and proposes a fix. See Diagnose a bad answer.
• Or check it yourself with the symptom table in How your bot answers. If the answer listed no sources, the bot didn't find anything relevant; if it listed the wrong ones, the search matched the wrong content.

Keep a simple list: the question, what went wrong and the cause. Patterns show up quickly, and three chats with the same cause need one fix, not three.

3. Fix the cause

Most fixes fall into one of these:

• Missing knowledge. Add the page to your website data source, or write the answer as a manual entry in your customers' own words. Manual entries are quick to add and easy to find later.
• Out-of-date knowledge. Re-sync the data source (Sync on its card, or Synchronise inside it), and turn on Auto-syncing if your content changes often. New pages on a website aren't found by a re-sync: add them. See Keep your bot's knowledge up to date.
• Instructions. Add a rule to the Prompt, such as what to do when the answer isn't in the knowledge, how long answers should be, or which topics are off-limits. Change one thing at a time. See Write your bot's instructions.
• Retrieval settings. Lower Document relevance or raise Max context documents if the bot misses content that's there; turn on Enhanced knowledge retrieval if it loses track of follow-up questions. See Retrieval settings.
• No way out. If customers get stuck, add Talk to a human or a contact form, so the bot can pass them on.

For the full list of fixes by problem, see Improve your bot's answers.

4. Check the fix worked

• Click Test on your bot's page and ask the original question, plus two or three rewordings. Chats you start here use message tokens like any other.
• On the Enterprise plan, save each fixed question as a test case with a check on the answer. Run your tests after every change so an old problem can't quietly come back.

5. Look at the bigger picture

Once you have a few weeks of conversations, step back from individual chats. On the Pro and Enterprise plans, your bot's Overview tab shows:

• Common Questions: the questions customers ask most. Ask each one yourself and check the answer. A frequent question with a weak answer is your most valuable fix.
• Topic Analysis: what people ask about, and how that changes week by week. A growing topic may need more content.
• Sentiment Analysis: whether customers are leaving happier or more frustrated over time.
• Actionable Recommendations: suggestions for your knowledge and instructions, based on patterns in your conversations.

Team members also get a weekly analytics email summarising each active bot. See Analytics.

On other plans, get a similar picture by downloading your conversations (Download all on the Conversations page gives a CSV of everything matching your filters) and skimming the questions.

Your weekly checklist

Once the first busy weeks are over, this takes 20 to 30 minutes a week:

1. In Conversations, filter for thumbs down, Escalated and (on Pro and Enterprise) negative sentiment since last week. Read each one.
2. Diagnose each bad answer with Ask Otto about this chat or the symptom table.
3. Fix the causes: add or update knowledge, adjust the Prompt or a retrieval setting.
4. Re-ask the questions you fixed with Test, or run your test cases on Enterprise.
5. Check your data sources synced recently and that no sync failed. Add any new pages from your site.
6. On Pro and Enterprise, scan Common Questions and Actionable Recommendations on the Overview tab.
7. Check your usage on the Overview tab so you don't run out of message tokens before the month ends.

Your bot doesn't learn from conversations on its own. Every improvement comes from a change you make to its knowledge, instructions or settings, which is why this routine matters.

Next steps

• How your bot answers: what goes into each reply, and the symptom table.
• Improve your bot's answers: step-by-step fixes by problem.
• Recipes for AI agents: have your AI assistant run the weekly review for you.