Let your bot read a web page

Add the Browse web page power-up so your bot can read a web page live during a conversation, for example a link the visitor shares, a page that changes too often to sync, or the top result from a Google search.

Who can do this: team owners and admins.

How it works

The bot decides what it wants from the page (its goal) and, unless you've fixed the URL, which page to open. Chat Thing fetches the page, keeps the part you selected, and uses your bot's model to pull out the information for that goal, or runs your custom prompt instead. The bot gets that result, not the whole page.

Video walkthrough; the second part covers Browse web page. Recorded in 2024, so some screens have changed since.

Steps

1. Open your bot, go to the Power-ups tab and click New power-up.
2. Choose Browse web page and click Create power-up.
3. Edit the Power-up name and Description. The description decides which pages the bot opens and when (see the examples below).
4. Under URL and Content, set:
   • URL: a fixed page the bot always reads, for example https://example.com/opening-hours. Leave blank to let the bot open any URL.
   • CSS selector: the part of the page to read, for example main or article. Leave blank to read the whole page body.
   • CSS excludes: parts of the page to skip. header and footer are excluded by default. Click the plus icon to add more, such as nav or .cookie-banner.
   • Basic auth: a Username and Password, only shown when you've set a fixed URL. See Pages behind a login.
5. Optionally turn on Use custom prompt? under Browsing prompt and write a Custom prompt (see below).
6. Optionally choose a Display mode to show the result as cards, a table or a chart. See Show a power-up's results visually.
7. Click Save.

Write the description

The description changes how the bot uses the power-up:

• Any page: "Read a web page when the user shares a link or asks about a specific page." With the URL blank, the bot can open any public page.
• One page, one job: "Use this to check today's opening hours and any closures." With the URL set to your opening hours page, the bot only reads that page, and only for that kind of question.

Use a custom prompt

By default, Chat Thing extracts whatever matches the bot's goal from the page. Turn on Use custom prompt? to control exactly what comes back instead. For example:

• "Extract all the blog posts from this page and return them as a table with the columns title, date and link."
• "Summarise this page in French."
• "List every product on this page with its price and stock status."

The custom prompt replaces the default instructions every time the power-up runs.

Pages behind a login

For a page protected by HTTP basic authentication, set a fixed https:// URL, then enter a Username and Password under Basic auth. The credentials are only ever sent to that fixed URL, never to a page the bot chooses. If the URL doesn't start with https://, the request fails rather than sending the credentials unencrypted.

Pages that need you to sign in through a form, or that use cookies or single sign-on, aren't supported.

Limits

• The power-up fetches the page's HTML. Content that only appears after JavaScript runs in a browser may be missing.
• Only web pages (HTML and XML) can be read. PDFs, images and other files are rejected with "Given URL is not a web page".
• Very large pages are rejected with "Page is too large to process" or "Page is too large to summarise". Use a CSS selector to read only the part you need.
• Slow pages can fail with "Request timed out".
• Only public internet addresses can be browsed.

Check it works

1. Chat with your bot while signed in and ask about the page, or share a link.
2. Open View details on the power-up call. The Power-up call shows the URL and goal, and the Power-up result shows what was extracted, or an error.

Troubleshooting

The result is empty or missing content

The content is probably loaded by JavaScript, or your CSS selector doesn't match anything on the page. Try a broader selector, or leave it blank to read the whole body.

The result includes menus, cookie notices or other clutter

Add them to CSS excludes, or set a CSS selector for the main content.

"The page returned an error" or "URL is not reachable"

The site returned an error, blocks automated requests, or the address isn't publicly reachable. Check that the page opens in a private browser window.