Storage tokens and how syncing uses them
Storage tokens measure how much content Chat Thing processes when it syncs your data sources. When you sync, Chat Thing splits your content into chunks and indexes them so your bot can search them. The tokens in that content are counted against your team's storage token allowance. 1,000 tokens is roughly 750 words.
Every plan includes a monthly storage token allowance, shared by every bot on your team. See Compare plans for each plan's allowance.
When syncing uses storage tokens
| Action | Storage tokens used |
|---|---|
| First sync of a new data source | All of its content |
| Re-sync (manual or automatic) | Only pages, files or rows whose content has changed since the last sync. Each changed item is counted in full |
| Re-sync with nothing changed | None |
| Chatting with your bot | None (chats use message tokens) |
| Deleting a data source | Doesn't give back tokens already used this month |
Storage tokens count content processed this month, not the total size of everything your bots know. Content you synced in earlier months stays searchable and doesn't count again unless it changes.
When the allowance resets
Your allowance resets on the 1st of each calendar month (UTC). Unused tokens don't roll over.
What happens when you reach the limit
- Your bots keep working. Content that's already synced stays searchable.
- New syncs are blocked. Starting a sync shows Over plan storage token limit, please upgrade plan.
- A sync that would take you over the limit doesn't start processing. It fails before any content is indexed, and the error tells you how many storage tokens the sync needs. Your existing content for that data source is kept.
- The data source shows a sync error, and team owners and admins get an email that the sync failed.
To sync again before the 1st, add a storage tokens add-on (5,000,000 tokens per unit) or upgrade your plan.
Use fewer storage tokens
- Only sync what your bot needs. Exclude pages that don't help answer questions. See Add a website.
- Scrape only the main content of each web page, leaving out headers, footers and menus. See Add a website.
- Don't add the same content twice, for example the same page in two data sources.
- Split large sources so you can re-sync only the part that changes.
For more, see Keep your bot up to date.
See your usage
- Sidebar: the Storage tokens bar shows how much of this month's allowance your team has used.
- My Bots: the Storage tokens this month card shows tokens used against your limit.
- A bot's Data sources tab: Storage tokens this month shows your team's usage this month, and Total storage tokens shows the current size of that bot's data sources.
Next steps
Related
- Message tokens and what happens when you run out
What message tokens are, how each message is counted, how model multipliers work, when tokens reset, and what happens when your team runs out.
- Keep your bot's knowledge up to date
Re-sync data sources by hand, on a daily, weekly or monthly schedule, or automatically when you publish in your CMS, and see what each sync changes.
- Add more tokens, bots or seats with add-ons
Raise one limit on your paid plan without upgrading - extra message tokens, storage tokens, bots, data sources, team members or custom domains.
Last updated