apify/crawlee · error · Error

Request object's uniqueKey must be a non-empty string

Error message

Request object's uniqueKey must be a non-empty string

What it means

ensureUniqueKeyValid() enforces that a request's uniqueKey is a non-empty string before it is stored or marked handled. A missing, null, undefined, or empty-string uniqueKey would corrupt the internal uniqueKeyToIndex map and in-progress tracking, so the library refuses it.

Source

Thrown at packages/core/src/storages/request_list.ts:779

        // Skip requests with duplicate uniqueKey
        if (!Object.hasOwn(this.#uniqueKeyToIndex, uniqueKey)) {
            this.#uniqueKeyToIndex[uniqueKey] = this.requests.length;
            this.requests.push(request);
        } else if (this.#keepDuplicateUrls) {
            this.#log.warning(
                `Duplicate uniqueKey: ${uniqueKey} found while the keepDuplicateUrls option was set. Check your sources' unique keys.`,
            );
        }
    }

    /**
     * Helper function that validates unique key.
     * Throws an error if uniqueKey is not a non-empty string.
     */
    private ensureUniqueKeyValid(uniqueKey: string): void {
        if (typeof uniqueKey !== 'string' || !uniqueKey) {
            throw new Error("Request object's uniqueKey must be a non-empty string");
        }
    }

    /**
     * Checks that a request is currently being processed and throws an error if not.
     */
    private ensureInProgress(uniqueKey: string): void {
        if (!this.inProgress.has(uniqueKey)) {
            throw new Error(`The request is not being processed (uniqueKey: ${uniqueKey})`);
        }
    }

    /**
     * Throws an error if request list wasn't initialized.
     */
    private ensureIsInitialized(): void {
        if (!this.#isInitialized) {
            throw new Error(

View on GitHub (pinned to dbe57fb09c)

Solutions

Always set uniqueKey via Request.computeUniqueKey(url) or let the library derive it from the URL
Check the request object actually has a non-empty uniqueKey before calling markRequestAsHandled
Use the Request returned from fetchNextRequest rather than hand-built objects when marking handled

Example fix

// before
await requestList.markRequestAsHandled({ url, uniqueKey: '' });
// after
import { Request } from 'crawlee';
await requestList.markRequestAsHandled({ url, uniqueKey: Request.computeUniqueKey({ url }) });

Defensive patterns

Strategy: validation

Validate before calling

function assertUniqueKey(req) {
  if (typeof req.uniqueKey !== 'string' || !req.uniqueKey) {
    req.uniqueKey = Request.computeUniqueKey({ url: req.url });
  }
}

Type guard

function hasValidUniqueKey(r) {
  return typeof r?.uniqueKey === 'string' && r.uniqueKey.length > 0;
}

Try / catch

try {
  await requestList.markRequestAsHandled(request);
} catch (err) {
  if (err.message.includes('uniqueKey must be a non-empty string')) {
    request.uniqueKey = Request.computeUniqueKey({ url: request.url });
    await requestList.markRequestAsHandled(request);
  } else throw err;
}

Prevention

Never hand-assign uniqueKey; let it be computed from the URL
Mark handled only with request objects returned by fetchNextRequest
Check JSON round-trips preserve the uniqueKey field

When it happens

Trigger: Calling markRequestAsHandled() with a request/uniqueKey that is empty or not a string; manually constructing Request-like objects with uniqueKey: '' and adding them via addRequest; overwriting request.uniqueKey to an empty value before adding.

Common situations: Copying request objects between queues and clearing uniqueKey; template string uniqueKeys that evaluate to ''; JSON round-trips dropping the field; passing the wrong variable to markRequestAsHandled.

Related errors

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/6abafb1d7f982271. Report an issue: GitHub.