Failed to load robots.txt from

Error message

Failed to load robots.txt from ${url}: HTTP ${response.status}

What it means

RobotsTxtFile.load fetches robots.txt via the given proxy/timeout options and requires an HTTP 2xx response. Any non-2xx status (500, 403, DNS-level failure pages, etc.) throws this error, except 404/other statuses handled as 'no robots.txt' cases — the throw here means the server explicitly returned an error status, so robots.txt policy cannot be determined.

Solutions

Check the HTTP status code in the message and fix the underlying server/WAF issue
Retry with a different proxy or residential proxy to avoid bot-blocking (403)
Add retry/backoff for transient 5xx responses
Verify the robots.txt URL scheme and host are correct
Decide on fallback behavior (treat as allow-all/disallow-all) in a try-catch around load()

Example fix

// before
const robots = await RobotsTxtFile.load(url);

// after
let robots;
try {
  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });
} catch (err) {
  log.warning(`robots.txt unavailable: ${err.message}; assuming allow-all`);
  robots = null;
}

Defensive patterns

Strategy: retry

Validate before calling

const head = await fetch(url, { method: 'HEAD' });
if (!head.ok && head.status !== 404) console.warn(`robots.txt may fail: HTTP ${head.status}`);

Type guard

function isOkStatus(s: number): boolean {
  return s >= 200 && s < 300;
}

Try / catch

try {
  robots = await RobotsTxtFile.load(url, { proxyUrl, timeoutMillis: 10_000 });
} catch (err) {
  if (String(err).includes('Failed to load robots.txt')) {
    robots = null; // fallback: treat as allow-all
  } else throw err;
}

Prevention

Use a residential/less-blocked proxy for robots.txt fetches (403 defense)
Retry transient 5xx with exponential backoff
Decide an explicit fallback policy when robots.txt is unavailable
Monitor HTTP statuses returned for robots.txt across your target domains

When it happens

Trigger: load() receives a response with status < 200 or >= 300 from the robots.txt URL — e.g. server returns 500, 403 (bot blocked by WAF), or an intermediary proxy returns an error page.

Common situations: Target site blocks datacenter IPs with 403 on robots.txt; origin server errors under load; CDN/WAF (Cloudflare) challenging the request; wrong scheme/port in the URL so an unexpected endpoint responds.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/671491433a0ca356. Report an issue: GitHub.

Appendix: source

Thrown at packages/utils/src/internals/robots.ts:110

        url: string,
        options?: {
            signal?: AbortSignal;
            timeoutMillis?: number;
            proxyUrl?: string;
            httpClient?: BaseHttpClient;
            logger?: CrawleeLogger;
        },
    ): Promise<RobotsTxtFile> {
        const { proxyUrl, logger, httpClient = new FetchHttpClient() } = options || {};

        const response = await httpClient.sendRequest(new Request(url, { method: 'GET' }), {
            proxyUrl,
            timeoutMillis: options?.timeoutMillis,
            signal: options?.signal,
        });

        if (response.status < 200 || response.status >= 300) {
            throw new Error(`Failed to load robots.txt from ${url}: HTTP ${response.status}`);
        }

        if (response.status === 404) {
            return new RobotsTxtFile(
                url,
                {
                    isAllowed() {
                        return true;
                    },
                    getSitemaps() {
                        return [];
                    },
                    getCrawlDelay() {
                        return undefined;
                    },
                },
                proxyUrl,
                logger,

View on GitHub (pinned to dbe57fb09c)