{"record":{"id":"4f252cb08467460d","repo":"apify/crawlee","slug":"cannot-fetch-a-request-list-from-requestsfromurl-4f252c","errorCode":null,"errorMessage":"Cannot fetch a request list from ${requestsFromUrl}: ${err}","messagePattern":"Cannot fetch a request list from (.+?): (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/storages/request_queue.ts","lineNumber":988,"sourceCode":"        return metadata;\n    }\n\n    /**\n     * Fetches URLs from requestsFromUrl and returns them in format of list of requests\n     */\n    private async fetchRequestsFromUrl(source: InternalSource): Promise<RequestOptions[]> {\n        const { requestsFromUrl, regex, ...sharedOpts } = source;\n\n        // Download remote resource and parse URLs.\n        let urlsArr;\n        try {\n            urlsArr = await this.downloadListOfUrls({\n                url: requestsFromUrl,\n                urlRegExp: regex,\n                proxyUrl: (await this.#proxyConfiguration?.newProxyInfo())?.url,\n            });\n        } catch (err) {\n            throw new Error(`Cannot fetch a request list from ${requestsFromUrl}: ${err}`);\n        }\n\n        // Skip if resource contained no URLs.\n        if (!urlsArr.length) {\n            this.log.warning('The fetched list contains no valid URLs.', { requestsFromUrl, regex });\n            return [];\n        }\n\n        return urlsArr.map((url) => ({ url, ...sharedOpts }));\n    }\n\n    /**\n     * Adds all fetched requests from a URL from a remote resource.\n     */\n    private async addFetchedRequests(\n        source: InternalSource,\n        fetchedRequests: RequestOptions[],\n        options: RequestQueueOperationOptions,","sourceCodeStart":970,"sourceCodeEnd":1006,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/storages/request_queue.ts#L970-L1006","documentation":"RequestQueue throws this when it fails to download a list of request URLs from a remote resource (`requestsFromUrl`, e.g. a text/CSV file hosted at a URL) via `downloadListOfUrls`. The underlying fetch/parse error `err` is appended to the message. It means the queue could not obtain the request list, so queueing from that remote source aborts.","triggerScenarios":"Calling RequestQueue.addRequests / queue operations with `requestsFromUrl` set while the remote URL is unreachable, returns a non-200 response, DNS fails, the configured proxy rejects the connection, the regex matches nothing due to unexpected content, or TLS fails.","commonSituations":"Hosted URL list rotated or deleted by the provider; wrong URL in crawler config; corporate proxy/firewall blocking the fetch; passing a proxyUrl whose credentials expired; typo in URL scheme (ftp:// instead of https://).","solutions":["Open the URL in a browser/curl and verify it is reachable and contains URLs","Check the appended `err` detail for DNS, TLS, proxy, or HTTP-status causes","Remove or fix proxyUrl if a stale proxy blocks the fetch","Download the list yourself and pass Request objects directly via addRequests"],"exampleFix":"// before\nawait queue.addRequests({ requestsFromUrl: 'http://old-host/urls.txt' });\n// after\nconst urls = await fs.readFile('urls.txt', 'utf8');\nawait queue.addRequests({ requests: urls.split('\\n').map((u) => ({ url: u.trim() })) });","handlingStrategy":"try-catch","validationCode":"const ok = await fetch(requestsFromUrl, { method: 'HEAD' }).then(r => r.ok).catch(() => false);\nif (!ok) throw new Error(`requestsFromUrl unreachable: ${requestsFromUrl}`);","typeGuard":"null","tryCatchPattern":"try {\n  await queue.addRequests({ requestsFromUrl });\n} catch (err) {\n  if (String(err).startsWith('Cannot fetch a request list from')) {\n    logger.warning('Falling back to local URL list', { cause: err });\n    return addLocalList();\n  }\n  throw err;\n}","preventionTips":["Verify requestsFromUrl with curl/HEAD before deploying","Pin stable, versioned URLs for hosted URL lists","Test proxy connectivity in CI","Prefer passing explicit request objects over remote lists when possible"],"tags":["network","http","request-queue"],"backgroundTag":"remote-url-fetch-failed","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}