apify/crawlee · error · Error

Cannot fetch a request list from ${requestsFromUrl}: ${err}

Error message

Cannot fetch a request list from ${requestsFromUrl}: ${err}

What it means

fetchRequestsFromUrl() downloads a remote list of URLs via downloadListOfUrls (optionally through a proxy). Any network, DNS, proxy, or HTTP failure is wrapped in this error naming the source URL and the underlying cause, so the real reason is in the message suffix after the colon.

Source

Thrown at packages/core/src/storages/request_list.ts:720

        return state!;
    }

    /**
     * Fetches URLs from requestsFromUrl and returns them in format of list of requests
     */
    private async fetchRequestsFromUrl(source: InternalSource): Promise<RequestOptions[]> {
        const { requestsFromUrl, regex, ...sharedOpts } = source;

        // Download remote resource and parse URLs.
        let urlsArr;
        try {
            urlsArr = await this.downloadListOfUrls({
                url: requestsFromUrl,
                urlRegExp: regex,
                proxyUrl: (await this.#proxyConfiguration?.newProxyInfo())?.url,
            });
        } catch (err) {
            throw new Error(`Cannot fetch a request list from ${requestsFromUrl}: ${err}`);
        }

        // Skip if resource contained no URLs.
        if (!urlsArr.length) {
            this.#log.warning('The fetched list contains no valid URLs.', { requestsFromUrl, regex });
            return [];
        }

        return urlsArr.map((url) => ({ url, ...sharedOpts }));
    }

    /**
     * Adds given request.
     * If the `source` parameter is a string or plain object and not an instance
     * of a `Request`, then the function creates a `Request` instance.
     */
    private addRequest(source: RequestListSource) {
        let request: Request | RequestOptions;

View on GitHub (pinned to dbe57fb09c)

Solutions

Read the underlying ${err} part of the message to identify the network cause (ENOTFOUND, 403, ECONNREFUSED, timeout)
Verify the URL is reachable (curl the URL, check status code and content-type)
Check/correct proxyConfiguration credentials and proxy availability, or test without a proxy
Add retry/timeout handling around list fetching and verify network egress in the execution environment

Example fix

// before
await RequestList.open('list', { requestsFromUrl: 'http://internal/list.txt' });
// after: validate reachability and handle failure
const res = await fetch('http://internal/list.txt');
if (!res.ok) throw new Error(`List URL unreachable: ${res.status}`);
await RequestList.open('list', { requestsFromUrl: 'http://internal/list.txt' });

Defensive patterns

Strategy: retry

Validate before calling

async function assertListUrlReachable(url) {
  const res = await fetch(url, { method: 'HEAD' });
  if (!res.ok) throw new Error(`List URL ${url} returned ${res.status}`);
}
await assertListUrlReachable(requestsFromUrl);

Type guard

null

Try / catch

try {
  await requestList.addRequestsFromUrl();
} catch (err) {
  if (err.message.startsWith('Cannot fetch a request list')) {
    logger.warning(`List fetch failed: ${err.message}`); // cause is after the colon
    await sleep(5000); // then retry, or fall back to a local list
  } else throw err;
}

Prevention

HEAD-check remote list URLs before runs
Verify proxy credentials and egress rules in the runtime environment
Pin list URLs and avoid links behind auth or expiring signatures

When it happens

Trigger: Calling RequestList.addRequestsFromUrl / initializing a list with requestsFromUrl when the URL is unreachable, returns non-2xx, DNS fails, the proxy (proxyConfiguration) rejects or times out, or TLS fails.

Common situations: Typo in the remote list URL; list hosted behind auth or a dead link; expired proxy credentials; firewall/egress blocking the host in CI; self-signed certificates.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/ab7ade4ed29574e0. Report an issue: GitHub.