apify/crawlee · error · Error

Loading requests with sourcesFunction failed. Cause: ${err.m

Error message

Loading requests with sourcesFunction failed.
Cause: ${err.message}

What it means

When sourcesFunction is invoked during initialization and it throws (network failure, bad fetch, parse error), the library cancels initialization and rethrows the cause wrapped in this prefixed error so you can tell which phase failed.

Source

Thrown at packages/core/src/storages/request_list.ts:448

        // Drop the original array full of empty indexes.
        this.#sources = [];

        if (this.#sourcesFunction) {
            try {
                const sourcesFromFunction = await this.#sourcesFunction();
                const sourcesFromFunctionCount = sourcesFromFunction.length;
                for (let i = 0; i < sourcesFromFunctionCount; i++) {
                    const source = sourcesFromFunction[i];
                    // oxlint-disable-next-line typescript/no-array-delete -- intentional, drop the slot so V8 can collect the object
                    delete sourcesFromFunction[i];
                    this.addRequest(source);
                }

                sourcesFromFunction.length = 0;
            } catch (e) {
                const err = e as Error;
                throw new Error(`Loading requests with sourcesFunction failed.\nCause: ${err.message}`);
            }
        }
    }

    /**
     * @inheritDoc
     */
    async persistState(): Promise<void> {
        if (!this.#persistStateKey) {
            throw new Error('Cannot persist state. options.persistStateKey is not set.');
        }
        if (this.isStatePersisted) return;
        try {
            this.#store ??= await KeyValueStore.open();
            await this.#store.setValue(this.#persistStateKey, this.getState());
            this.isStatePersisted = true;
        } catch (e) {
            const err = e as Error;

View on GitHub (pinned to dbe57fb09c)

Solutions

Fix the underlying error reported after 'Cause:' (check network/endpoint/auth)
Make sourcesFunction defensive: retry fetching, validate the response before returning
Provide static sources as a fallback when the dynamic source is unavailable

Example fix

// before
sourcesFunction: async () => (await fetch(url)).json(),
// after
sourcesFunction: async () => {
    for (let i = 0; i < 3; i++) {
        try {
            const res = await fetch(url);
            if (!res.ok) throw new Error(`HTTP ${res.status}`);
            return res.json();
        } catch (e) {
            if (i === 2) throw e;
            await new Promise((r) => setTimeout(r, 2000));
        }
    }
}

Defensive patterns

Strategy: retry

Try / catch

try {
  const list = await RequestList.open('list', { sourcesFunction: loadUrls });
} catch (e) {
  if (e.message.startsWith('Loading requests with sourcesFunction failed')) {
    console.error('sourcesFunction failed:', e.message.split('Cause:')[1]);
    // retry with backoff or fall back to static sources
  } else throw e;
}

Prevention

Wrap the fetch logic inside sourcesFunction with its own retries and timeouts
Validate the fetched payload shape before returning it as sources
Provide a static sources fallback for resilience

When it happens

Trigger: sourcesFunction returns a promise that rejects — e.g. fetching URLs from an API that returns 500, DNS failure, invalid response shape that throws while mapping, or a synchronous throw inside the function body.

Common situations: Loading URL lists from a Google Sheets export, internal API, or sitemap whose endpoint is down or rate-limiting; auth token expired for the sources API; deploying to an environment without network access to the source.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/13bb7e51567807cd. Report an issue: GitHub.