Knowledge Base
Web Crawling
Crawl websites to automatically add web content to your knowledge base.
Overview
Instead of uploading files manually, you can crawl a website and automatically import its content into your assistant's knowledge base.
How to Crawl a Website
- Go to Knowledge Base tab
- Click Add from URL (or Web Crawl)
- Enter the website URL (e.g.,
https://example.com/docs) - Configure crawl settings:
- Max pages — Limit the number of pages to crawl
- Depth — How many link levels deep to follow
- Click Start Crawl
- Pages are converted to text and processed like uploaded documents
Web crawling is powered by Firecrawl — an AI-optimized web scraping service that extracts clean, readable content from any website.
Best Use Cases
- Company documentation or help center
- Blog posts and articles
- Product pages and FAQs
- Public knowledge bases
Only crawl websites you have permission to use. Respect robots.txt and terms of service.