The website source is the fastest way to get answers flowing: give BeforeQuery a documentation URL and it crawls, cleans, and indexes every page. It works out of the box with the popular docs frameworks — Docusaurus, Mintlify, GitBook, ReadMe, VitePress, MkDocs, Hugo, and Next.js — which all ship pre-rendered HTML. A site that builds its pages in the browser is the exception: the crawler reads the HTML your server returns and does not run JavaScript.

​
Data ingested

When you crawl a website, BeforeQuery indexes:
  • Page URL — what every citation links back to
  • Page title, from <title> or the first <h1>
  • Page content as markdown — headings, lists (numbering kept), tables, blockquotes, and code blocks with their language hint
  • Heading hierarchy, so each chunk carries the section it came from
  • Image alt text, kept inline
Navigation, headers, footers, sidebars and aria-hidden elements are removed before conversion, and relative links are resolved to absolute URLs. Images themselves are not read — only their alt text. Set Also index linked files to index the documents your pages link to as well — a datasheet PDF, a Word template. Each becomes its own document, extracted exactly as an upload of the same file would be, and cited at its own URL. Empty indexes HTML pages only.

​
Set up a website source

1

Add the source

In the dashboard, open Knowledge Base → Sources → Add source and choose Website, then enter your docs URL (e.g. https://docs.example.com). If you created the knowledge base from a docs URL, this source already exists and the first crawl ran automatically.
2

Watch the crawl

The crawler discovers your sitemap.xml for complete coverage, respects robots.txt, strips navigation chrome, preserves fenced code blocks with language hints, and converts every page to clean markdown before chunking and embedding. Progress is shown on the source card; from there you can also cancel a running crawl.
3

Verify in the Playground

Ask a question your docs answer. Citations should deep-link to the right page section — chunking is heading-aware, so answers map to your document structure.

​
What the crawler does

  • Sitemap discovery — finds and uses sitemap.xml for complete coverage.
  • robots.txt compliance — never fetches disallowed paths.
  • Navigation stripping — removes navbars, sidebars, and footers so answers come from content, not chrome.
  • Code block extraction — answers include runnable snippets.
  • Incremental sync — content-hash change detection re-embeds only what changed.
  • Linked files — indexes PDFs and other documents your pages link to, when you name their extensions.
  • Outbound proxy — crawl through your own address when a site blocks ours, or when a fixed IP has to be allowlisted.

​
Review and curate indexed pages

Open Knowledge Base → Documents to browse everything the crawl produced. From a document’s page you can:
  • Inspect the markdown BeforeQuery extracted — useful when an answer cites something unexpected.
  • Delete a page (and its chunks) that shouldn’t be answerable.
  • Restrict a page so it answers only for workspace members, never public traffic.
Mark internal-only pages restricted to keep them out of widget, SDK, and bot retrieval while keeping them available to your team — see Roles & Permissions.

​
Keep the site in sync

A website source re-crawls every 24 hours, and there is no schedule to configure. It is the slowest cadence in the product for a reason: a docs site cannot be asked what changed since yesterday, so a refresh walks every page in scope. Each page is asked instead — the crawler sends back the ETag and Last-Modified your server gave it last time, and a page your server reports unchanged is never read, chunked or embedded. Pages behind a server that sends neither are caught by the content hash instead. You do not have to wait for it. Sync re-crawls the whole source now, and each run only re-processes pages whose content hash changed, so a pass after a docs release stays cheap. For a single page, the source’s page list gives you two narrower controls:
  • Add page — index a URL the crawl will never reach, such as an orphan with no inbound links. It is pinned, so later crawls keep it.
  • Re-crawl this page — refetch the one doc you just published, without touching the other nine hundred.
Neither counts as a whole-site pass, so neither pushes the scheduled crawl further out. See Sources Overview.

​
Troubleshooting

  • Pages missing? Check they’re reachable from your sitemap and not disallowed by robots.txt.
  • Answers citing stale content? Re-crawl the single page from the source’s page list, or trigger a full Sync, then check the source’s last synced time. Automatic re-crawls run every 24 hours.
  • Marketing pages polluting answers? Delete them from Documents, or crawl only your docs subdomain rather than the whole site.
  • Linked PDFs not indexed? Add their extension to Also index linked files. Without it the crawler fetches the link, sees it is not HTML, and skips it — the source’s page list shows those as skipped.
  • Every page failing with 403? The site is refusing us rather than failing. Bot protection judges the address a request comes from, and ours is a data centre — nothing in the request itself changes that verdict. Set an Outbound proxy to crawl through a different address.
  • Site behind an IP allowlist? Point Outbound proxy at a proxy with a fixed address and give your IT team that address to allow.
  • Site renders in the browser? The crawler does not run JavaScript. Point it at a pre-rendered build, or add the pages by URL from the source’s page list.