---
title: "Train Your AI Agent on 100+ Pages in 2 Minutes"
canonical_url: "https://chatthing.ai/blog/website-crawling"
last_updated: "2026-10-01T15:28:55.699Z"
meta:
description: "Stop adding pages one by one. Chat Thing can crawl your entire website and train your AI agent automatically. Set it up in 2 minutes."
"og:description": "Stop adding pages one by one. Chat Thing can crawl your entire website and train your AI agent automatically. Set it up in 2 minutes."
"og:title": "Website Crawling: Train Your Agent on 100+ Pages in 2 Minutes - Blog - Chat Thing"
"twitter:description": "Stop adding pages one by one. Chat Thing can crawl your entire website and train your AI agent automatically. Set it up in 2 minutes."
"twitter:title": "Website Crawling: Train Your Agent on 100+ Pages in 2 Minutes - Blog - Chat Thing"
---
[← Back to blog](https://chatthing.ai/blog)**Blog**# **Website Crawling: Train Your Agent on 100+ Pages in 2 Minutes**
Gem
30 Mar 2026
---
---
### **On this page**
1. [The old way vs the new way](#the-old-way-vs-the-new-way)
2. [How to crawl your website](#how-to-crawl-your-website)
3. [Pro tip: Use content selectors](#pro-tip-use-content-selectors)
4. [What about sites behind a login?](#what-about-sites-behind-a-login)
5. [When crawling makes sense](#when-crawling-makes-sense)
6. [Keep it fresh with auto-sync](#keep-it-fresh-with-auto-sync)
7. [Go try it](#go-try-it)
# How to Train Your Agent on 100+ Pages in 2 Minutes
You want your agent to know your entire website. But adding pages one by one? Nobody has time for that.
Maybe you've got a docs site with 200 pages. Maybe your help centre keeps growing. Maybe you've been putting this off because it sounds like a nightmare.
Good news: Chat Thing can crawl your entire site and add it to your agent automatically.
## The old way vs the new way
The old way:
Copy URL. Paste. Click add. Repeat. Repeat. Repeat. Give up after 15 pages and hope nobody asks about the other 185.
The new way:
Give Chat Thing one URL. Click "Crawl". Go make a coffee. Come back to an agent that knows your entire website.
We'll discover up to 600 pages automatically, let you review them, and sync the lot in one go.
## How to crawl your website
1. Go to your agent's [tab](https://chatthing.ai/app/bots) (see documentation [here](https://chatthing.ai/docs/website-data-source)).
2. Click New data source and select Website
3. Click the Crawl button
4. Enter your website's URL and hit Crawl
5. Wait while we discover your pages (larger sites take a few minutes)
6. Review the discovered URLs and toggle off any you don't want
7. Click Add URLs and then Synchronise
That's it. Your agent now knows your entire site.
## Pro tip: Use content selectors
By default, we grab everything in the page body (minus headers and footers, which would just waste tokens by repeating on every page).
But you can get smarter. If your content lives in a specific container, like main or .article-content, set that as your CSS selector. Your agent gets cleaner data, and you use fewer storage tokens.
Worth spending 5 minutes on before you sync 200 pages.
## What about sites behind a login?
If your site uses Basic Auth (that browser popup asking for username/password), we've got you covered. Just add your credentials in the scraping settings and we'll access the protected content. Other login types (like form-based logins or OAuth) aren't supported though - we can only crawl public pages or Basic Auth protected ones.
## When crawling makes sense
Great for:
- Documentation sites
- Help centres and FAQs
- Marketing sites with lots of pages
- Blogs and content libraries
- Any site where pages link to each other
Maybe not ideal for:
- Single-page apps (crawling won't find much)
- Sites with lots of duplicate content
- Pages behind form logins or OAuth (Basic Auth is fine)
## Keep it fresh with auto-sync
Once you've crawled your site, turn on auto-sync to keep it up to date. We'll check for new and changed pages on whatever schedule you set.
Your agent stays current without you lifting a finger.
## Go try it
Got a website with more than a handful of pages? Give crawling a try. It takes about 2 minutes to set up, and your agent gets instant access to everything.
[Create a free account](https://chatthing.ai/app/auth/register) and add your first website data source. Your future self will thank you.
### **Related articles **
[](https://chatthing.ai/blog/chat-thing-webmcp-website-copilot) [Turn a WebMCP-enabled website into a copilot with Chat Thing
](https://chatthing.ai/blog/chat-thing-webmcp-website-copilot)
Chat Thing can now discover the tools a WebMCP-enabled page exposes and let an embedded agent use them. Add the widget, switch on WebMCP, and the page has a conversational copilot.
[](https://chatthing.ai/blog/supportbench) [SupportBench: we tested 7 AI models as customer support agents
](https://chatthing.ai/blog/supportbench)
We tested seven AI models on 1,085 support conversations. Gemini 3.7 Flash led the main score, while Grok 4.6 wrote the preferred reply most often.
[](https://chatthing.ai/blog/blog-coworking-space-agent) [Our co-working space down the road just got an AI agent. Here's what happened.
](https://chatthing.ai/blog/blog-coworking-space-agent)
C-Space is a coastal coworking space in Newquay with hot desks, private offices and event hire. This case study looks at how their AI agent answers the everyday questions, parking, opening hours, walk-ins, the moment they come in, so the team can stay focused on running the space instead of sitting on the inbox.
- **Previous **
- [← Auto-sync: Your Bot Never Gets Stale](https://chatthing.ai/blog/auto-sync-your-bot-never-gets-stale)
- **Next **
- [HR Agent: The Internal Knowledge Agent Your Team Will Actually Use →](https://chatthing.ai/blog/hr-agent-internal-knowledge-agent-your-team-will-use)