apify/crawlee · error · Error

RequestList sources are already loading or were loaded.

Error message

RequestList sources are already loading or were loaded.

What it means

initialize() loads remote sources and starts persistence; running it twice on the same RequestList would duplicate requests and corrupt state. A guard flag (#isLoading) makes a second call throw this error.

Source

Thrown at packages/core/src/storages/request_list.ts:360

        this.#keepDuplicateUrls = keepDuplicateUrls;

        // Will be empty after initialization to save memory.
        this.#sources = sources ? [...sources] : [];
        this.#sourcesFunction = sourcesFunction;

        // The proxy configuration used for `requestsFromUrl` requests.
        this.#proxyConfiguration = proxyConfiguration;

        this.persistState = this.persistState.bind(this);
    }

    /**
     * Loads all remote sources of URLs and potentially starts periodic state persistence.
     * This function must be called before you can start using the instance in a meaningful way.
     */
    private async initialize(): Promise<this> {
        if (this.#isLoading) {
            throw new Error('RequestList sources are already loading or were loaded.');
        }

        this.#isLoading = true;
        await purgeDefaultStorages({ onlyPurgeOnce: true });

        const [state, persistedRequests] = await this.loadStateAndPersistedRequests();

        // Add persisted requests / new sources in a memory efficient way because with very
        // large lists, we were running out of memory.
        if (persistedRequests) {
            await this.addPersistedRequests(persistedRequests as Buffer);
        } else {
            await this.addRequestsFromSources();
        }

        this.restoreState(state as RequestListState);
        this.#isInitialized = true;
        if (this.#persistRequestsKey && !this.areRequestsPersisted) await this.persistRequests();

View on GitHub (pinned to dbe57fb09c)

Solutions

Call RequestList.open() only — it initializes internally; remove manual initialize() calls
Cache the returned instance and reuse it instead of re-opening/re-initializing
Use a memoized promise for open() if it may be invoked concurrently from several places

Example fix

// before
const list = await RequestList.open('list', opts);
await list.initialize(); // throws
// after
const list = await RequestList.open('list', opts); // already initialized

Defensive patterns

Strategy: validation

Validate before calling

const listCache = new Map();
async function openListOnce(name, options) {
  if (!listCache.has(name)) listCache.set(name, RequestList.open(name, options));
  return listCache.get(name);
}

Try / catch

try {
  await list.initialize();
} catch (e) {
  if (e.message.includes('already loading or were loaded')) {
    // safe to ignore: list is ready or initialization is in progress
  } else throw e;
}

Prevention

Rely on RequestList.open() — never call initialize() manually
Memoize open() promises to prevent concurrent double-open
Reuse a single list instance across your crawler setup

When it happens

Trigger: Calling list.initialize() twice; RequestList.open() called twice for the same list while an earlier initialization is in-flight or completed internally; manual initialize after open() (open() already initializes).

Common situations: Calling RequestList.open() and then also list.initialize() in legacy migrated code; retry/failed-run wrappers that re-open the list without checking; concurrent open() calls from multiple code paths.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/0f1488f94d205679. Report an issue: GitHub.