{"record":{"id":"671491433a0ca356","repo":"apify/crawlee","slug":"failed-to-load-robots-txt-from-url-http-resp","errorCode":null,"errorMessage":"Failed to load robots.txt from ${url}: HTTP ${response.status}","messagePattern":"Failed to load robots\\.txt from (.+?): HTTP (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/utils/src/internals/robots.ts","lineNumber":110,"sourceCode":"        url: string,\n        options?: {\n            signal?: AbortSignal;\n            timeoutMillis?: number;\n            proxyUrl?: string;\n            httpClient?: BaseHttpClient;\n            logger?: CrawleeLogger;\n        },\n    ): Promise<RobotsTxtFile> {\n        const { proxyUrl, logger, httpClient = new FetchHttpClient() } = options || {};\n\n        const response = await httpClient.sendRequest(new Request(url, { method: 'GET' }), {\n            proxyUrl,\n            timeoutMillis: options?.timeoutMillis,\n            signal: options?.signal,\n        });\n\n        if (response.status < 200 || response.status >= 300) {\n            throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);\n        }\n\n        if (response.status === 404) {\n            return new RobotsTxtFile(\n                url,\n                {\n                    isAllowed() {\n                        return true;\n                    },\n                    getSitemaps() {\n                        return [];\n                    },\n                    getCrawlDelay() {\n                        return undefined;\n                    },\n                },\n                proxyUrl,\n                logger,","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/utils/src/internals/robots.ts#L92-L128","documentation":"RobotsTxtFile.load fetches robots.txt via the given proxy/timeout options and requires an HTTP 2xx response. Any non-2xx status (500, 403, DNS-level failure pages, etc.) throws this error, except 404/other statuses handled as 'no robots.txt' cases — the throw here means the server explicitly returned an error status, so robots.txt policy cannot be determined.","triggerScenarios":"load() receives a response with status < 200 or >= 300 from the robots.txt URL — e.g. server returns 500, 403 (bot blocked by WAF), or an intermediary proxy returns an error page.","commonSituations":"Target site blocks datacenter IPs with 403 on robots.txt; origin server errors under load; CDN/WAF (Cloudflare) challenging the request; wrong scheme/port in the URL so an unexpected endpoint responds.","solutions":["Check the HTTP status code in the message and fix the underlying server/WAF issue","Retry with a different proxy or residential proxy to avoid bot-blocking (403)","Add retry/backoff for transient 5xx responses","Verify the robots.txt URL scheme and host are correct","Decide on fallback behavior (treat as allow-all/disallow-all) in a try-catch around load()"],"exampleFix":"// before\nconst robots = await RobotsTxtFile.load(url);\n\n// after\nlet robots;\ntry {\n  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });\n} catch (err) {\n  log.warning(`robots.txt unavailable: ${err.message}; assuming allow-all`);\n  robots = null;\n}","handlingStrategy":"retry","validationCode":"const head = await fetch(url, { method: 'HEAD' });\nif (!head.ok && head.status !== 404) console.warn(`robots.txt may fail: HTTP ${head.status}`);","typeGuard":"function isOkStatus(s: number): boolean {\n  return s >= 200 && s < 300;\n}","tryCatchPattern":"try {\n  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });\n} catch (err) {\n  if (String(err).includes('Failed to load robots.txt')) {\n    robots = null; // fallback: treat as allow-all\n  } else throw err;\n}","preventionTips":["Use a residential/less-blocked proxy for robots.txt fetches (403 defense)","Retry transient 5xx with exponential backoff","Decide an explicit fallback policy when robots.txt is unavailable","Monitor HTTP statuses returned for robots.txt across your target domains"],"tags":["network","robots-txt","http"],"backgroundTag":"http-error-status","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}