{"record":{"id":"ab7ade4ed29574e0","repo":"apify/crawlee","slug":"cannot-fetch-a-request-list-from-requestsfromurl","errorCode":null,"errorMessage":"Cannot fetch a request list from ${requestsFromUrl}: ${err}","messagePattern":"Cannot fetch a request list from (.+?): (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/request_list.ts","lineNumber":720,"sourceCode":"        return state!;\n    }\n\n    /**\n     * Fetches URLs from requestsFromUrl and returns them in format of list of requests\n     */\n    private async fetchRequestsFromUrl(source: InternalSource): Promise<RequestOptions[]> {\n        const { requestsFromUrl, regex, ...sharedOpts } = source;\n\n        // Download remote resource and parse URLs.\n        let urlsArr;\n        try {\n            urlsArr = await this.downloadListOfUrls({\n                url: requestsFromUrl,\n                urlRegExp: regex,\n                proxyUrl: (await this.#proxyConfiguration?.newProxyInfo())?.url,\n            });\n        } catch (err) {\n            throw new Error(`Cannot fetch a request list from ${requestsFromUrl}: ${err}`);\n        }\n\n        // Skip if resource contained no URLs.\n        if (!urlsArr.length) {\n            this.#log.warning('The fetched list contains no valid URLs.', { requestsFromUrl, regex });\n            return [];\n        }\n\n        return urlsArr.map((url) => ({ url, ...sharedOpts }));\n    }\n\n    /**\n     * Adds given request.\n     * If the `source` parameter is a string or plain object and not an instance\n     * of a `Request`, then the function creates a `Request` instance.\n     */\n    private addRequest(source: RequestListSource) {\n        let request: Request | RequestOptions;","sourceCodeStart":702,"sourceCodeEnd":738,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/request_list.ts#L702-L738","documentation":"fetchRequestsFromUrl() downloads a remote list of URLs via downloadListOfUrls (optionally through a proxy). Any network, DNS, proxy, or HTTP failure is wrapped in this error naming the source URL and the underlying cause, so the real reason is in the message suffix after the colon.","triggerScenarios":"Calling RequestList.addRequestsFromUrl / initializing a list with requestsFromUrl when the URL is unreachable, returns non-2xx, DNS fails, the proxy (proxyConfiguration) rejects or times out, or TLS fails.","commonSituations":"Typo in the remote list URL; list hosted behind auth or a dead link; expired proxy credentials; firewall/egress blocking the host in CI; self-signed certificates.","solutions":["Read the underlying ${err} part of the message to identify the network cause (ENOTFOUND, 403, ECONNREFUSED, timeout)","Verify the URL is reachable (curl the URL, check status code and content-type)","Check/correct proxyConfiguration credentials and proxy availability, or test without a proxy","Add retry/timeout handling around list fetching and verify network egress in the execution environment"],"exampleFix":"// before\nawait RequestList.open('list', { requestsFromUrl: 'http://internal/list.txt' });\n// after: validate reachability and handle failure\nconst res = await fetch('http://internal/list.txt');\nif (!res.ok) throw new Error(`List URL unreachable: ${res.status}`);\nawait RequestList.open('list', { requestsFromUrl: 'http://internal/list.txt' });","handlingStrategy":"retry","validationCode":"async function assertListUrlReachable(url) {\n  const res = await fetch(url, { method: 'HEAD' });\n  if (!res.ok) throw new Error(`List URL ${url} returned ${res.status}`);\n}\nawait assertListUrlReachable(requestsFromUrl);","typeGuard":"null","tryCatchPattern":"try {\n  await requestList.addRequestsFromUrl();\n} catch (err) {\n  if (err.message.startsWith('Cannot fetch a request list')) {\n    logger.warning(`List fetch failed: ${err.message}`); // cause is after the colon\n    await sleep(5000); // then retry, or fall back to a local list\n  } else throw err;\n}","preventionTips":["HEAD-check remote list URLs before runs","Verify proxy credentials and egress rules in the runtime environment","Pin list URLs and avoid links behind auth or expiring signatures"],"tags":["network","http","request-list"],"backgroundTag":"http-request-failed","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}