Summary
In discoverValidSitemaps (@crawlee/utils), when robots.txt already yields Sitemap: URLs, the fallback HEAD-probing of the guessed candidates /sitemap.xml, /sitemap.txt, /sitemap_index.xml still appears to run — firing up to 3 redundant HEAD requests per domain. Guard the candidate-probing so it is skipped when robots.txt (or the URL-is-sitemap check) already produced sitemap URLs.
Detail
Observed while reading the compiled source of @crawlee/utils@3.17.1-beta.47 (internals/sitemap.js:369-467), functions discoverValidSitemaps / discoverSitemapsForDomainUrls. Discovery order:
- robots.txt first (
:410-418) — RobotsFile.find(...) → getSitemaps(), yields Sitemap: directive URLs.
- URL-is-sitemap check (
:423-428) — regex /sitemap\.(?:xml|txt)(?:\.gz)?$/i.
- HEAD-probe guessed candidates (
:429-450) — ['/sitemap.xml','/sitemap.txt','/sitemap_index.xml'] via urlExists (method: 'HEAD').
The candidate-probing in step 3 appears to fire even when step 1 already returned sitemap URLs, adding up to 3 unnecessary HEAD requests per domain. (Please confirm against current main — this was read from a beta build's compiled JS.)
Impact: minor per-domain latency/network waste, but multiplied across high-volume sitemap crawlers (e.g. Apify's Website Content Crawler runs millions of useSitemaps runs; robots.txt-declared sitemaps are common).
Proposed change
- Only run the guessed-candidate HEAD probing when robots.txt and the URL-is-sitemap check produced no sitemap URLs.
- File:
packages/utils/src/internals/sitemap.ts (discoverValidSitemaps / discoverSitemapsForDomainUrls).
Notes
- Minor optimization, surfaced while analyzing WCC sitemap behaviour. The rest of the discovery path is already efficient (robots.txt-first, HEAD-only probes, time-boxed, graceful no-sitemap handling).
- Out of scope: soft-404 servers (200 on nonexistent paths) can yield false-positive candidates, but those fail gracefully at the later GET+parse.
Surfaced during a downstream (WCC) sitemap analysis. 🤖 Generated with Claude Code
Summary
In
discoverValidSitemaps(@crawlee/utils), whenrobots.txtalready yieldsSitemap:URLs, the fallback HEAD-probing of the guessed candidates/sitemap.xml,/sitemap.txt,/sitemap_index.xmlstill appears to run — firing up to 3 redundant HEAD requests per domain. Guard the candidate-probing so it is skipped when robots.txt (or the URL-is-sitemap check) already produced sitemap URLs.Detail
Observed while reading the compiled source of
@crawlee/utils@3.17.1-beta.47(internals/sitemap.js:369-467), functionsdiscoverValidSitemaps/discoverSitemapsForDomainUrls. Discovery order::410-418) —RobotsFile.find(...)→getSitemaps(), yieldsSitemap:directive URLs.:423-428) — regex/sitemap\.(?:xml|txt)(?:\.gz)?$/i.:429-450) —['/sitemap.xml','/sitemap.txt','/sitemap_index.xml']viaurlExists(method: 'HEAD').The candidate-probing in step 3 appears to fire even when step 1 already returned sitemap URLs, adding up to 3 unnecessary HEAD requests per domain. (Please confirm against current
main— this was read from a beta build's compiled JS.)Impact: minor per-domain latency/network waste, but multiplied across high-volume sitemap crawlers (e.g. Apify's Website Content Crawler runs millions of
useSitemapsruns; robots.txt-declared sitemaps are common).Proposed change
packages/utils/src/internals/sitemap.ts(discoverValidSitemaps/discoverSitemapsForDomainUrls).Notes
Surfaced during a downstream (WCC) sitemap analysis. 🤖 Generated with Claude Code