apify/crawlee · error · Error

Request ID does not match its uniqueKey.

Error message

Request ID does not match its uniqueKey.

What it means

The request queue stores requests keyed by a uniqueKey, and derives the internal request ID deterministically from that uniqueKey via uniqueKeyToRequestId. When a caller supplies a RequestSchema whose explicit `id` does not match the ID recomputed from its `uniqueKey`, the queue refuses the write because the stored record would be inconsistent (lookup by id and by uniqueKey would disagree).

Source

Thrown at packages/core/src/memory-storage/resource-clients/request-queue.ts:495

        this.accessedAt = new Date();

        if (hasBeenModified) {
            this.modifiedAt = new Date();
        }
    }

    private jsonToRequest<T>(requestJson?: string): T | undefined {
        if (!requestJson) return undefined;
        const request = JSON.parse(requestJson);
        return purgeNullsFromObject(request);
    }

    private createInternalRequest(request: storage.RequestSchema, forefront?: boolean): InternalRequest {
        const orderNo = this.calculateOrderNo(request, forefront);
        const id = uniqueKeyToRequestId(request.uniqueKey);

        if (request.id && request.id !== id) {
            throw new Error('Request ID does not match its uniqueKey.');
        }

        const json = JSON.stringify({ ...request, id });
        return {
            id,
            json,
            method: request.method,
            orderNo,
            retryCount: request.retryCount ?? 0,
            uniqueKey: request.uniqueKey,
            url: request.url,
        };
    }

    private calculateOrderNo(request: storage.RequestSchema, forefront?: boolean) {
        if (request.handledAt) return null;

        const timestamp = Date.now();

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Remove the explicit `id` field and let createInternalRequest compute it from the uniqueKey.
2. Set `id` to the value returned by uniqueKeyToRequestId(request.uniqueKey).
3. If the uniqueKey changed, recompute the id (or drop it) before calling addRequest.
4. Verify you are not reusing request objects across queues with different uniqueKey normalization.

Example fix

// before
await queue.addRequest({ id: 'abc-123', uniqueKey: 'https://example.com', url: 'https://example.com' });
// after
await queue.addRequest({ uniqueKey: 'https://example.com', url: 'https://example.com' }); // id derived automatically

import { uniqueKeyToRequestId } from '@crawlee/core';
if (request.id && request.id !== uniqueKeyToRequestId(request.uniqueKey)) {
  throw new Error(`id ${request.id} does not match uniqueKey ${request.uniqueKey}`);
}

try {
  await queue.addRequest(request);
} catch (err) {
  if ((err as Error).message.includes('does not match its uniqueKey')) {
    delete request.id;
    await queue.addRequest(request);
  } else throw err;
}

When it happens

Trigger: Calling queue.addRequest() (via requestModel -> createInternalRequest) with a request object that has both `id` and `uniqueKey` set, where `id !== uniqueKeyToRequestId(uniqueKey)` — e.g. reusing an ID from a different request, or hand-crafting a request with an id not derived from the uniqueKey.

Common situations: Manually constructing Request objects with copied/stale IDs; migrating requests between queues while keeping old ids; generating uniqueKey after assigning id; deserializing requests whose uniqueKey was changed but id was not recomputed.

Related errors

• ${operation} cannot be used inside a storage transaction: ${
• The `requestManager` option cannot be used in conjunction wi
• ServiceConflictError('StorageBackend', storageBackend, this.
• Request options are not valid, the 'url' property is not a s
• Request options are not valid, the 'id' property must not be

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/9bfa4434f7801145. Report an issue: GitHub.