apify/crawlee · error · Error

Cannot fetch a request list from ${requestsFromUrl}: ${err}

Error message

Cannot fetch a request list from ${requestsFromUrl}: ${err}

What it means

RequestQueue throws this when it fails to download a list of request URLs from a remote resource (`requestsFromUrl`, e.g. a text/CSV file hosted at a URL) via `downloadListOfUrls`. The underlying fetch/parse error `err` is appended to the message. It means the queue could not obtain the request list, so queueing from that remote source aborts.

Source

Thrown at packages/core/src/storages/request_queue.ts:988

        return metadata;
    }

    /**
     * Fetches URLs from requestsFromUrl and returns them in format of list of requests
     */
    private async fetchRequestsFromUrl(source: InternalSource): Promise<RequestOptions[]> {
        const { requestsFromUrl, regex, ...sharedOpts } = source;

        // Download remote resource and parse URLs.
        let urlsArr;
        try {
            urlsArr = await this.downloadListOfUrls({
                url: requestsFromUrl,
                urlRegExp: regex,
                proxyUrl: (await this.#proxyConfiguration?.newProxyInfo())?.url,
            });
        } catch (err) {
            throw new Error(`Cannot fetch a request list from ${requestsFromUrl}: ${err}`);
        }

        // Skip if resource contained no URLs.
        if (!urlsArr.length) {
            this.log.warning('The fetched list contains no valid URLs.', { requestsFromUrl, regex });
            return [];
        }

        return urlsArr.map((url) => ({ url, ...sharedOpts }));
    }

    /**
     * Adds all fetched requests from a URL from a remote resource.
     */
    private async addFetchedRequests(
        source: InternalSource,
        fetchedRequests: RequestOptions[],
        options: RequestQueueOperationOptions,

View on GitHub (pinned to dbe57fb09c)

Solutions

Open the URL in a browser/curl and verify it is reachable and contains URLs
Check the appended `err` detail for DNS, TLS, proxy, or HTTP-status causes
Remove or fix proxyUrl if a stale proxy blocks the fetch
Download the list yourself and pass Request objects directly via addRequests

Example fix

// before
await queue.addRequests({ requestsFromUrl: 'http://old-host/urls.txt' });
// after
const urls = await fs.readFile('urls.txt', 'utf8');
await queue.addRequests({ requests: urls.split('\n').map((u) => ({ url: u.trim() })) });

Defensive patterns

Strategy: try-catch

Validate before calling

const ok = await fetch(requestsFromUrl, { method: 'HEAD' }).then(r => r.ok).catch(() => false);
if (!ok) throw new Error(`requestsFromUrl unreachable: ${requestsFromUrl}`);

Type guard

null

Try / catch

try {
  await queue.addRequests({ requestsFromUrl });
} catch (err) {
  if (String(err).startsWith('Cannot fetch a request list from')) {
    logger.warning('Falling back to local URL list', { cause: err });
    return addLocalList();
  }
  throw err;
}

Prevention

Verify requestsFromUrl with curl/HEAD before deploying
Pin stable, versioned URLs for hosted URL lists
Test proxy connectivity in CI
Prefer passing explicit request objects over remote lists when possible

When it happens

Trigger: Calling RequestQueue.addRequests / queue operations with `requestsFromUrl` set while the remote URL is unreachable, returns a non-200 response, DNS fails, the configured proxy rejects the connection, the regex matches nothing due to unexpected content, or TLS fails.

Common situations: Hosted URL list rotated or deleted by the provider; wrong URL in crawler config; corporate proxy/firewall blocking the fetch; passing a proxyUrl whose credentials expired; typo in URL scheme (ftp:// instead of https://).

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/4f252cb08467460d. Report an issue: GitHub.