apify/crawlee · error · Error

The request is not being processed (url: ${url})

Error message

The request is not being processed (url: ${url})

What it means

SitemapRequestLoader.enforceInProgress is an internal invariant check: it is thrown when markRequestAsHandled (or a similar completion API) is invoked for a URL that the loader does not currently have marked as in-progress. It signals a lifecycle violation — the request was never started, was already handled, or the loader state was reset.

Source

Thrown at packages/core/src/storages/sitemap_request_loader.ts:635

        this.#events.off(EventType.PERSIST_STATE, this.persistState);
        await this.persistState();

        this.#urlQueueStream.emit('readdata'); // unblocks the potentially waiting `pushNextUrl` call
    }

    /**
     * @inheritDoc
     */
    async markRequestAsHandled(request: Request): Promise<void> {
        this.#handledUrlCount += 1;
        this.ensureInProgress(request.url);
        this.inProgress.delete(request.url);
        this.#requestData.delete(request.url);
    }

    private ensureInProgress(url: string): void {
        if (!this.inProgress.has(url)) {
            throw new Error(`The request is not being processed (url: ${url})`);
        }
    }
}

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Only call markRequestAsHandled for requests returned by the loader's own fetch/next flow
2. Guard against double-handling by tracking processed URLs in your code
3. Re-add the request to the queue instead of marking an unknown URL as handled
4. Check for loader reset (reinit) between starting and handling requests

Example fix

// before
loader.markRequestAsHandled(request); // may throw if not started
// after
try {
  loader.markRequestAsHandled(request);
} catch (err) {
  if (!String(err).includes('not being processed')) throw err;
  loader.inProgressRequests?.add(request.url); // or re-fetch/restart handling
}

null

null

try {
  loader.markRequestAsHandled(request);
} catch (err) {
  if (!/not being processed/.test(String(err))) throw err;
  logger.warning('Attempted to handle untracked sitemap request', { url: request.url });
}

When it happens

Trigger: Calling markRequestAsHandled with a request whose URL was never started by the loader, marking the same URL handled twice, or handling a request after the sitemap loader was reinitialized/cleared.

Common situations: Custom crawlers manually calling markRequestAsHandled outside the loader's normal processing loop; re-enqueueing a handled request object; concurrency bugs where two workers process the same sitemap URL.

Related errors

• StateValidationError(persistStateKey, result.issues)
• Recoverable state has not yet been loaded - call initialize(
• Cannot clear the state persisted under key '${this.#persistS
• The state object is invalid: nextIndex must be a non-negativ
• The state object is not consistent with RequestList, too few

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/501fc82634073134. Report an issue: GitHub.