ID's must be unique, found duplicates for: ${duplicateIds}

Error message

ID's must be unique, found duplicates for: ${duplicateIds}

What it means

Thrown client-side by prepareRecordRequest in chromadb-core before any request is sent, when the ids array passed to collection.add(), collection.upsert(), or collection.update() contains duplicate values. Chroma uses ids as primary keys inside a collection, so one batch with repeated ids is ambiguous (is it an insert or an update?) and is rejected up front. The message lists the duplicated id values so you can locate them.

Source

Thrown at clients/js/packages/chromadb-core/src/utils.ts:153

    }
  }

  if (
    (embeddingsArray !== undefined && ids.length !== embeddingsArray.length) ||
    (metadatas !== undefined && ids.length !== metadatas.length) ||
    (documents !== undefined && ids.length !== documents.length)
  ) {
    throw new Error(
      "ids, embeddings, metadatas, and documents must all be the same length",
    );
  }

  const uniqueIds = new Set(ids);
  if (uniqueIds.size !== ids.length) {
    const duplicateIds = ids.filter(
      (item, index) => ids.indexOf(item) !== index,
    );
    throw new Error(
      `ID's must be unique, found duplicates for: ${duplicateIds}`,
    );
  }

  if (
    embeddingsArray &&
    embeddingsArray.some((embedding) => embedding.length === 0)
  ) {
    throw new Error("got empty embedding at pos");
  }

  return {
    ids,
    metadatas,
    documents,
    embeddings: embeddingsArray,
  };
}

View on GitHub (pinned to aecdd12c8a)

Solutions

Inspect the duplicated ids in the message and remove or regenerate them so every id in the batch is unique (e.g. crypto.randomUUID() per record).
If the records already exist and you want to overwrite them, use collection.upsert() instead of add() — but still pass each id only once per call.
Deduplicate the input at the source: new Set(ids) size check, or key records by a guaranteed-unique column before batching.
If duplicates come from resumable ingestion, track already-ingested ids and skip them before the next add().

Example fix

// before
await collection.add({
  ids: ["doc1", "doc1", "doc2"],
  documents: ["a", "a-dup", "b"],
});

// after
await collection.add({
  ids: ["doc1", "doc1-dup", "doc2"], // or dedupe first
  documents: ["a", "a-dup", "b"],
});

// or overwrite existing rows in one pass
await collection.upsert({
  ids: ["doc1", "doc2"],
  documents: ["a", "b"],
});

Defensive patterns

Strategy: validation

Validate before calling

function assertUniqueIds(ids: string[]): void {
  const seen = new Set<string>();
  const dupes: string[] = [];
  for (const id of ids) {
    if (seen.has(id)) dupes.push(id);
    seen.add(id);
  }
  if (dupes.length > 0) {
    throw new Error(`Duplicate ids in batch: ${[...new Set(dupes)].join(", ")}`);
  }
}

assertUniqueIds(ids);
await collection.add({ ids, documents });

Prevention

Generate ids with crypto.randomUUID() unless you control uniqueness at the source.
Run new Set(ids).size === ids.length before every add()/upsert() batch.
In resumable ingestion, persist already-sent ids and filter them out of later batches.
Prefer upsert() when records may already exist, but still send each id once per call.

When it happens

Trigger: Calling collection.add({ ids: ["a", "a", "b"] }) or upsert/update with a repeated id; concatenating result sets (e.g. DB cursor pages or chunked files) without deduplicating; building ids from a row counter that resets between batches; passing a content-hash id when the same document appears twice in one batch.

Common situations: Ingest pipelines that resume from a cursor and re-read overlapping rows; looping over pages that share boundary records; adding the same records twice in one call by mistake; generating ids with a hash function that collides or with a non-unique source column.

Related errors

AI-assisted analysis of chroma-core/chroma@aecdd12c8a (2026-08-16). Data as JSON: /api/errors/ec095834dfe461e1. Report an issue: GitHub.