The website source is the fastest way to get answers flowing: give BeforeQuery a documentation URL and it crawls, cleans, and indexes every page. It works out of the box with JavaScript-rendered sites and all popular docs frameworks — Docusaurus, Mintlify, GitBook, ReadMe, VitePress, MkDocs, Hugo, and Next.js sites.

​
Set up a website source

1

Add the source

In the dashboard, open Project → Sources → Add source and choose Website, then enter your docs URL (e.g. https://docs.example.com). If you created the project from a docs URL, this source already exists and the first crawl ran automatically.
2

Watch the crawl

The crawler discovers your sitemap.xml for complete coverage, respects robots.txt, strips navigation chrome, preserves fenced code blocks with language hints, and converts every page to clean markdown before chunking and embedding. Progress is shown on the source card; from there you can also cancel a running crawl.
3

Verify in the Playground

Ask a question your docs answer. Citations should deep-link to the right page section — chunking is heading-aware, so answers map to your document structure.

​
What the crawler does

  • Sitemap discovery — finds and uses sitemap.xml for complete coverage.
  • robots.txt compliance — never fetches disallowed paths.
  • Navigation stripping — removes navbars, sidebars, and footers so answers come from content, not chrome.
  • Code block extraction — answers include runnable snippets.
  • Incremental sync — content-hash change detection re-embeds only what changed.

​
Review and curate indexed pages

Open Project → Documents to browse everything the crawl produced. From a document’s page you can:
  • Inspect the markdown BeforeQuery extracted — useful when an answer cites something unexpected.
  • Delete a page (and its chunks) that shouldn’t be answerable.
  • Restrict a page so it answers only for workspace members, never public traffic.
Mark internal-only pages restricted to keep them out of widget, SDK, and bot retrieval while keeping them available to your team — see Roles & Permissions.

​
Keep the site in sync

Set the source’s sync schedule to manual, daily, or weekly on the source card, and use the Sync button after big docs releases. Each sync only re-processes pages whose content hash changed, so frequent syncs stay cheap.

​
Troubleshooting

  • Pages missing? Check they’re reachable from your sitemap and not disallowed by robots.txt.
  • Answers citing stale content? Trigger a manual Sync — then check the source’s last synced time to confirm your schedule is frequent enough.
  • Marketing pages polluting answers? Delete them from Documents, or crawl only your docs subdomain rather than the whole site.