Train Your AI Agent on 100+ Pages in 2 Minutes
[Chat Thing](https://chatthing.ai/)
Open main menu
- Features
- Use Cases
- [Pricing](https://chatthing.ai/pricing)
- Resources
[Log in](https://app.chatthing.ai/app/auth/login) [ Book a demo](https://chatthing.ai/demo) [Sign up](https://app.chatthing.ai/app/auth/register)
[← Back to blog](https://chatthing.ai/blog)Blog # Website Crawling: Train Your Agent on 100+ Pages in 2 Minutes
Gem
30 Mar 2026
---
---
### On this page
1. [The old way vs the new way](#the-old-way-vs-the-new-way)
2. [How to crawl your website](#how-to-crawl-your-website)
3. [Pro tip: Use content selectors](#pro-tip-use-content-selectors)
4. [What about sites behind a login?](#what-about-sites-behind-a-login)
5. [When crawling makes sense](#when-crawling-makes-sense)
6. [Keep it fresh with auto-sync](#keep-it-fresh-with-auto-sync)
7. [Go try it](#go-try-it)
# How to Train Your Agent on 100+ Pages in 2 Minutes
You want your agent to know your entire website. But adding pages one by one? Nobody has time for that.
Maybe you've got a docs site with 200 pages. Maybe your help centre keeps growing. Maybe you've been putting this off because it sounds like a nightmare.
Good news: Chat Thing can crawl your entire site and add it to your agent automatically.
## The old way vs the new way
The old way:
Copy URL. Paste. Click add. Repeat. Repeat. Repeat. Give up after 15 pages and hope nobody asks about the other 185.
The new way:
Give Chat Thing one URL. Click "Crawl". Go make a coffee. Come back to an agent that knows your entire website.
We'll discover up to 600 pages automatically, let you review them, and sync the lot in one go.
## How to crawl your website
1. Go to your agent's [tab](https://chatthing.ai/app/bots) (see documentation [here](https://chatthing.ai/docs/website-data-source?utm_source=blog&utm_medium=content&utm_campaign=website-crawling)).
2. Click New data source and select Website
3. Click the Crawl button
4. Enter your website's URL and hit Crawl
5. Wait while we discover your pages (larger sites take a few minutes)
6. Review the discovered URLs and toggle off any you don't want
7. Click Add URLs and then Synchronise
That's it. Your agent now knows your entire site.
## Pro tip: Use content selectors
By default, we grab everything in the page body (minus headers and footers, which would just waste tokens by repeating on every page).
But you can get smarter. If your content lives in a specific container, like main or .article-content, set that as your CSS selector. Your agent gets cleaner data, and you use fewer storage tokens.
Worth spending 5 minutes on before you sync 200 pages.
## What about sites behind a login?
If your site uses Basic Auth (that browser popup asking for username/password), we've got you covered. Just add your credentials in the scraping settings and we'll access the protected content. Other login types (like form-based logins or OAuth) aren't supported though - we can only crawl public pages or Basic Auth protected ones.
## When crawling makes sense
Great for:
- Documentation sites
- Help centres and FAQs
- Marketing sites with lots of pages
- Blogs and content libraries
- Any site where pages link to each other
Maybe not ideal for:
- Single-page apps (crawling won't find much)
- Sites with lots of duplicate content
- Pages behind form logins or OAuth (Basic Auth is fine)
## Keep it fresh with auto-sync
Once you've crawled your site, turn on auto-sync to keep it up to date. We'll check for new and changed pages on whatever schedule you set.
Your agent stays current without you lifting a finger.
## Go try it
Got a website with more than a handful of pages? Give crawling a try. It takes about 2 minutes to set up, and your agent gets instant access to everything.
[Create a free account](https://chatthing.ai/app/auth/register?utm_source=blog&utm_medium=content&utm_campaign=website-crawling) and add your first website data source. Your future self will thank you.
### Related articles
[](https://chatthing.ai/blog/use-api) [Connect Your Agent to Any API
](https://chatthing.ai/blog/use-api)
Sometimes the built-in power-ups aren't enough. The Use API power-up lets your agent call any REST API directly, so it can check hotel availability, pull live weather data, or query your own internal systems, no pre-built integration required.
[](https://chatthing.ai/blog/bot-to-bot-power-up) [Build Multi-Agent AI Teams with Agent-to-Agent
](https://chatthing.ai/blog/bot-to-bot-power-up)
Why settle for one generalist agent when you can build a team of specialists? The Agent-to-Agent power-up lets your agents delegate tasks, ask each other questions, and combine expertise, so complex requests get handled by the right expert every time.
[](https://chatthing.ai/blog/email-power-up) [Capture Leads with AI - The Email Power-up
](https://chatthing.ai/blog/email-power-up)
Every conversation your website agent has is a missed opportunity if it just disappears. The Email power-up changes that, letting your agent capture visitor details, flag bugs, and send conversation summaries straight to your team's inbox, no forms and no friction required.
- Previous
- [← Auto-sync: Your Bot Never Gets Stale](https://chatthing.ai/blog/auto-sync-your-bot-never-gets-stale)
- Next
- [HR Agent: The Internal Knowledge Agent Your Team Will Actually Use →](https://chatthing.ai/blog/hr-agent-internal-knowledge-agent-your-team-will-use)
## Footer
### Ready to answer every question, day or night?
Add an AI support agent to your site in minutes.
[Get started free](https://app.chatthing.ai/app/auth/register) [Book a demo](https://chatthing.ai/demo)
AI customer support that learns from your content and answers 24/7.
### Features
- [Channels: Deploy your bot](https://chatthing.ai/pages/features/channels)
- [Data sources: Add knowledge](https://chatthing.ai/pages/features/data-sources)
- [Power-ups: Tool calling](https://chatthing.ai/pages/features/power-ups)
- [AI Models: GPT-4, Claude, & more](https://chatthing.ai/pages/features/models)
- [SDK: Create custom assistants](https://chatthing.ai/pages/features/chatbot-sdk)
- [Bot evaluations: Test your bot](https://chatthing.ai/pages/features/testing)
- [Analytics: Topics, sentiment & insights](https://chatthing.ai/pages/features/analytics)
- [Bot Gallery: Example bots](https://chatthing.ai/gallery)
- [All features](https://chatthing.ai/features)
### Use cases
- [AI Customer Support Bots](https://chatthing.ai/pages/use-cases/customer-support)
- [AI Internal Knowledge Bots](https://chatthing.ai/pages/use-cases/internal-knowledge)
- [AI Bots for Marketing](https://chatthing.ai/pages/use-cases/marketing)
- [SDK: Create custom assistants](https://chatthing.ai/pages/features/chatbot-sdk)
- [All features](https://chatthing.ai/features)
### Channels
- [AI support for your website](https://chatthing.ai/pages/features/channels/website)
- [AI support for Slack](https://chatthing.ai/pages/features/channels/slack)
- [AI support for Discord](https://chatthing.ai/pages/features/channels/discord)
- [AI support for WhatsApp](https://chatthing.ai/pages/features/channels/whatsapp)
- [AI support for Telegram](https://chatthing.ai/pages/features/channels/telegram)
- [AI support API](https://chatthing.ai/pages/features/channels/api)
### Data sources
- [Chat with your website](https://chatthing.ai/pages/data-source/web)
- [Chat with your files](https://chatthing.ai/pages/data-source/files)
- [Notion chatbots](https://chatthing.ai/pages/data-source/notion)
- [Chat with your youtube](https://chatthing.ai/pages/data-source/youtube)
- [Chat with your RSS feed](https://chatthing.ai/pages/data-source/rss)
### Company
- [Discord](https://discord.com/invite/DGCCR34Fsj)
- [Docs](https://chatthing.ai/docs/getting-started)
- [Blog](https://chatthing.ai/blog)
- [FAQ](https://chatthing.ai/faq)
- [Status](https://status.chatthing.ai/)
- [Testimonials](https://senja.io/p/chat-thing/iq1tWz)
- [Custom AI development](https://chatthing.ai/pages/ai-development-services)
### Free tools
- [Agent Readiness Checker](https://chatthing.ai/tools/agent-readiness-checker)
- [JSON-LD Generator + Checker](https://chatthing.ai/tools/json-ld-generator)
- [llms.txt Generator & Validator](https://chatthing.ai/tools/llms-txt-generator)
- [Support Ticket Deflection Calculator](https://chatthing.ai/tools/ticket-deflection-calculator)
- [Knowledge Base Grader](https://chatthing.ai/tools/knowledge-base-grader)
- [AI FAQ Generator](https://chatthing.ai/tools/faq-generator)
- [AI Canned Response Generator](https://chatthing.ai/tools/canned-response-generator)
- [Support Bot Prompt Generator](https://chatthing.ai/tools/support-bot-prompt-generator)
- [All free tools](https://chatthing.ai/tools)
### Compare
- [Intercom alternative](https://chatthing.ai/pages/alternatives/intercom)
- [Zendesk alternative](https://chatthing.ai/pages/alternatives/zendesk)
- [Tidio alternative](https://chatthing.ai/pages/alternatives/tidio)
- [Crisp alternative](https://chatthing.ai/pages/alternatives/crisp)
- [Chatbase alternative](https://chatthing.ai/pages/alternatives/chatbase)
- [DocsBot alternative](https://chatthing.ai/pages/alternatives/docsbot)
- [ResolveAI alternative](https://chatthing.ai/pages/alternatives/resolveai)
- [ChatBotKit alternative](https://chatthing.ai/pages/alternatives/chatbotkit)
- [CustomGPT alternative](https://chatthing.ai/pages/alternatives/customgpt)
- [Botsonic alternative](https://chatthing.ai/pages/alternatives/botsonic)
- [Ada alternative](https://chatthing.ai/pages/alternatives/ada)
### Join our newsletter
Be the first to know about new features and updates.
Join
[Privacy](https://chatthing.ai/legal/privacy) [Terms](https://chatthing.ai/legal/terms) [DPA](https://chatthing.ai/legal/dpa) [GDPR](https://chatthing.ai/legal/gdpr) [Sub-Processors](https://chatthing.ai/legal/sub-processors)
[Twitter](https://twitter.com/pixelhopio) [Discord](https://discord.com/invite/DGCCR34Fsj) [YouTube](https://www.youtube.com/@ChatThingAI)
© 2026 Chat Thing. All rights reserved. Company number: 14789358. Built with by [Pixelhop](https://pixelhop.io?utm_source=chat+thing&utm_medium=website)
## Cookies
We use cookies to run the site and understand how it's used. You can change your choice anytime.
Accept
Decline
Customise