apify/crawlee · error

Failed to load robots.txt from ${url}: HTTP ${response.statu

Error message

Failed to load robots.txt from ${url}: HTTP ${response.status}

What it means

RobotsTxtFile.load fetches robots.txt via the given proxy/timeout options and requires an HTTP 2xx response. Any non-2xx status (500, 403, DNS-level failure pages, etc.) throws this error, except 404/other statuses handled as 'no robots.txt' cases — the throw here means the server explicitly returned an error status, so robots.txt policy cannot be determined.

Source

Thrown at packages/utils/src/internals/robots.ts:110

        url: string,
        options?: {
            signal?: AbortSignal;
            timeoutMillis?: number;
            proxyUrl?: string;
            httpClient?: BaseHttpClient;
            logger?: CrawleeLogger;
        },
    ): Promise<RobotsTxtFile> {
        const { proxyUrl, logger, httpClient = new FetchHttpClient() } = options || {};

        const response = await httpClient.sendRequest(new Request(url, { method: 'GET' }), {
            proxyUrl,
            timeoutMillis: options?.timeoutMillis,
            signal: options?.signal,
        });

        if (response.status < 200 || response.status >= 300) {
            throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);
        }

        if (response.status === 404) {
            return new RobotsTxtFile(
                url,
                {
                    isAllowed() {
                        return true;
                    },
                    getSitemaps() {
                        return [];
                    },
                    getCrawlDelay() {
                        return undefined;
                    },
                },
                proxyUrl,
                logger,

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Check the HTTP status code in the message and fix the underlying server/WAF issue
2. Retry with a different proxy or residential proxy to avoid bot-blocking (403)
3. Add retry/backoff for transient 5xx responses
4. Verify the robots.txt URL scheme and host are correct
5. Decide on fallback behavior (treat as allow-all/disallow-all) in a try-catch around load()

Example fix

// before
const robots = await RobotsTxtFile.load(url);

// after
let robots;
try {
  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });
} catch (err) {
  log.warning(`robots.txt unavailable: ${err.message}; assuming allow-all`);
  robots = null;
}

const head = await fetch(url, { method: 'HEAD' });
if (!head.ok && head.status !== 404) console.warn(`robots.txt may fail: HTTP ${head.status}`);

function isOkStatus(s: number): boolean {
  return s >= 200 && s < 300;
}

try {
  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });
} catch (err) {
  if (String(err).includes('Failed to load robots.txt')) {
    robots = null; // fallback: treat as allow-all
  } else throw err;
}

When it happens

Trigger: load() receives a response with status < 200 or >= 300 from the robots.txt URL — e.g. server returns 500, 403 (bot blocked by WAF), or an intermediary proxy returns an error page.

Common situations: Target site blocks datacenter IPs with 403 on robots.txt; origin server errors under load; CDN/WAF (Cloudflare) challenging the request; wrong scheme/port in the URL so an unexpected endpoint responds.

Related errors

• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• Too many redirects (${maxRedirects}) while requesting ${curr
• Request timed out after ${this.#navigationTimeoutMillis / 10
• Skipping request ${request.url} as disallowed by robots.txt

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/671491433a0ca356. Report an issue: GitHub.