{"record":{"id":"c9ac075fa5e5c448","repo":"apify/crawlee","slug":"the-provided-newurlfunction-did-not-return-a-valid","errorCode":null,"errorMessage":"The provided newUrlFunction did not return a valid URL.\nCause: ${(err as Error).message}","messagePattern":"The provided newUrlFunction did not return a valid URL\\.\nCause: (.+?)","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/core/src/proxy_configuration.ts","lineNumber":189,"sourceCode":"        return this.handleProxyUrlsList() ?? undefined;\n    }\n\n    private handleProxyUrlsList(): string | null {\n        return this.#proxyUrls![this.#nextCustomUrlIndex++ % this.#proxyUrls!.length];\n    }\n\n    /**\n     * Calls the custom newUrlFunction and checks format of its return value\n     */\n    private async callNewUrlFunction(options?: { request?: Request }) {\n        const proxyUrl = await this.#newUrlFunction!(options);\n        try {\n            if (proxyUrl) {\n                new URL(proxyUrl); // eslint-disable-line no-new\n            }\n            return proxyUrl;\n        } catch (err) {\n            throw new Error(\n                `The provided newUrlFunction did not return a valid URL.\\nCause: ${(err as Error).message}`,\n            );\n        }\n    }\n\n    private throwCannotCombineCustomMethods(): never {\n        throw new Error(\n            'Cannot combine custom proxies \"options.proxyUrls\" with custom generating function \"options.newUrlFunction\".',\n        );\n    }\n\n    private throwNoOptionsProvided(): never {\n        throw new Error('One of \"options.proxyUrls\" or \"options.newUrlFunction\" needs to be provided.');\n    }\n}\n","sourceCodeStart":171,"sourceCodeEnd":205,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/core/src/proxy_configuration.ts#L171-L205","documentation":"When ProxyConfiguration uses a newUrlFunction, the returned string must be a parseable proxy URL. callNewUrlFunction validates the result with `new URL(proxyUrl)`; if that fails (or the function returned null/undefined/empty), it rethrows with this message and the underlying URL-parse error as the cause.","triggerScenarios":"newUrlFunction returned an empty string, null/undefined, a non-URL string (e.g. 'localhost:8080' without scheme is actually valid for URL, but 'not a url' is not), or threw internally, and the proxy was then requested for a new session URL.","commonSituations":"Custom newUrlFunction reading env vars that are unset or malformed; returning a host without scheme and port mangling; a function that returns undefined on some code path; rotating-URL services returning a URL with characters needing encoding.","solutions":["Fix the newUrlFunction to always return a complete, valid proxy URL (include scheme, e.g. http://).","Handle empty/unset inputs inside the function and fall back to a known-good URL or throw a clear error.","Log the returned value when the error occurs to see what was actually produced.","Validate with `new URL(value)` inside your function before returning, to fail with a clearer message."],"exampleFix":"// before\nconst proxy = new ProxyConfiguration({ newUrlFunction: () => process.env.PROXY_URL }); // may be undefined\n// after\nconst proxy = new ProxyConfiguration({\n  newUrlFunction: () => {\n    const url = process.env.PROXY_URL;\n    if (!url) throw new Error('PROXY_URL is not set');\n    new URL(url); // validate early\n    return url;\n  },\n});","handlingStrategy":"validation","validationCode":"function returnsValidProxyUrl(fn) {\n  const v = fn();\n  try { new URL(v); return true; } catch { return false; }\n}\nif (!returnsValidProxyUrl(myNewUrlFunction)) throw new Error('newUrlFunction must return a valid URL');","typeGuard":"function isValidUrl(v: unknown): v is string {\n  if (typeof v !== 'string' || v.length === 0) return false;\n  try { new URL(v); return true; } catch { return false; }\n}","tryCatchPattern":"try {\n  await crawler.run(urls);\n} catch (err) {\n  if ((err as Error).message.includes('newUrlFunction did not return a valid URL')) {\n    console.error('Check your newUrlFunction output and env vars:', (err as Error).cause);\n  }\n  throw err;\n}","preventionTips":["Always include the scheme (http:// or https://) in generated proxy URLs.","Fail fast inside newUrlFunction when inputs are missing.","Unit-test newUrlFunction against empty/unset env vars.","URL-encode credentials in proxy URLs."],"tags":["proxy","url-validation","configuration"],"backgroundTag":"invalid-proxy-url","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}