apify/crawlee · error · Error

The provided newUrlFunction did not return a valid URL. Caus

Error message

The provided newUrlFunction did not return a valid URL.
Cause: ${(err as Error).message}

What it means

When ProxyConfiguration uses a newUrlFunction, the returned string must be a parseable proxy URL. callNewUrlFunction validates the result with `new URL(proxyUrl)`; if that fails (or the function returned null/undefined/empty), it rethrows with this message and the underlying URL-parse error as the cause.

Source

Thrown at packages/core/src/proxy_configuration.ts:189

        return this.handleProxyUrlsList() ?? undefined;
    }

    private handleProxyUrlsList(): string | null {
        return this.#proxyUrls![this.#nextCustomUrlIndex++ % this.#proxyUrls!.length];
    }

    /**
     * Calls the custom newUrlFunction and checks format of its return value
     */
    private async callNewUrlFunction(options?: { request?: Request }) {
        const proxyUrl = await this.#newUrlFunction!(options);
        try {
            if (proxyUrl) {
                new URL(proxyUrl); // eslint-disable-line no-new
            }
            return proxyUrl;
        } catch (err) {
            throw new Error(
                `The provided newUrlFunction did not return a valid URL.\nCause: ${(err as Error).message}`,
            );
        }
    }

    private throwCannotCombineCustomMethods(): never {
        throw new Error(
            'Cannot combine custom proxies "options.proxyUrls" with custom generating function "options.newUrlFunction".',
        );
    }

    private throwNoOptionsProvided(): never {
        throw new Error('One of "options.proxyUrls" or "options.newUrlFunction" needs to be provided.');
    }
}

View on GitHub (pinned to dbe57fb09c)

Solutions

  1. Fix the newUrlFunction to always return a complete, valid proxy URL (include scheme, e.g. http://).
  2. Handle empty/unset inputs inside the function and fall back to a known-good URL or throw a clear error.
  3. Log the returned value when the error occurs to see what was actually produced.
  4. Validate with `new URL(value)` inside your function before returning, to fail with a clearer message.

Example fix

// before
const proxy = new ProxyConfiguration({ newUrlFunction: () => process.env.PROXY_URL }); // may be undefined
// after
const proxy = new ProxyConfiguration({
  newUrlFunction: () => {
    const url = process.env.PROXY_URL;
    if (!url) throw new Error('PROXY_URL is not set');
    new URL(url); // validate early
    return url;
  },
});
Defensive patterns

Strategy: validation

Validate before calling

function returnsValidProxyUrl(fn) {
  const v = fn();
  try { new URL(v); return true; } catch { return false; }
}
if (!returnsValidProxyUrl(myNewUrlFunction)) throw new Error('newUrlFunction must return a valid URL');

Type guard

function isValidUrl(v: unknown): v is string {
  if (typeof v !== 'string' || v.length === 0) return false;
  try { new URL(v); return true; } catch { return false; }
}

Try / catch

try {
  await crawler.run(urls);
} catch (err) {
  if ((err as Error).message.includes('newUrlFunction did not return a valid URL')) {
    console.error('Check your newUrlFunction output and env vars:', (err as Error).cause);
  }
  throw err;
}

Prevention

When it happens

Trigger: newUrlFunction returned an empty string, null/undefined, a non-URL string (e.g. 'localhost:8080' without scheme is actually valid for URL, but 'not a url' is not), or threw internally, and the proxy was then requested for a new session URL.

Common situations: Custom newUrlFunction reading env vars that are unset or malformed; returning a host without scheme and port mangling; a function that returns undefined on some code path; rotating-URL services returning a URL with characters needing encoding.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/c9ac075fa5e5c448. Report an issue: GitHub.