{"record":{"id":"ec095834dfe461e1","repo":"chroma-core/chroma","slug":"id-s-must-be-unique-found-duplicates-for-dupli","errorCode":null,"errorMessage":"ID's must be unique, found duplicates for: ${duplicateIds}","messagePattern":"ID's must be unique, found duplicates for: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"clients/js/packages/chromadb-core/src/utils.ts","lineNumber":153,"sourceCode":"    }\n  }\n\n  if (\n    (embeddingsArray !== undefined && ids.length !== embeddingsArray.length) ||\n    (metadatas !== undefined && ids.length !== metadatas.length) ||\n    (documents !== undefined && ids.length !== documents.length)\n  ) {\n    throw new Error(\n      \"ids, embeddings, metadatas, and documents must all be the same length\",\n    );\n  }\n\n  const uniqueIds = new Set(ids);\n  if (uniqueIds.size !== ids.length) {\n    const duplicateIds = ids.filter(\n      (item, index) => ids.indexOf(item) !== index,\n    );\n    throw new Error(\n      `ID's must be unique, found duplicates for: ${duplicateIds}`,\n    );\n  }\n\n  if (\n    embeddingsArray &&\n    embeddingsArray.some((embedding) => embedding.length === 0)\n  ) {\n    throw new Error(\"got empty embedding at pos\");\n  }\n\n  return {\n    ids,\n    metadatas,\n    documents,\n    embeddings: embeddingsArray,\n  };\n}","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/clients/js/packages/chromadb-core/src/utils.ts#L135-L171","documentation":"Thrown client-side by prepareRecordRequest in chromadb-core before any request is sent, when the ids array passed to collection.add(), collection.upsert(), or collection.update() contains duplicate values. Chroma uses ids as primary keys inside a collection, so one batch with repeated ids is ambiguous (is it an insert or an update?) and is rejected up front. The message lists the duplicated id values so you can locate them.","triggerScenarios":"Calling collection.add({ ids: [\"a\", \"a\", \"b\"] }) or upsert/update with a repeated id; concatenating result sets (e.g. DB cursor pages or chunked files) without deduplicating; building ids from a row counter that resets between batches; passing a content-hash id when the same document appears twice in one batch.","commonSituations":"Ingest pipelines that resume from a cursor and re-read overlapping rows; looping over pages that share boundary records; adding the same records twice in one call by mistake; generating ids with a hash function that collides or with a non-unique source column.","solutions":["Inspect the duplicated ids in the message and remove or regenerate them so every id in the batch is unique (e.g. crypto.randomUUID() per record).","If the records already exist and you want to overwrite them, use collection.upsert() instead of add() — but still pass each id only once per call.","Deduplicate the input at the source: new Set(ids) size check, or key records by a guaranteed-unique column before batching.","If duplicates come from resumable ingestion, track already-ingested ids and skip them before the next add()."],"exampleFix":"// before\nawait collection.add({\n  ids: [\"doc1\", \"doc1\", \"doc2\"],\n  documents: [\"a\", \"a-dup\", \"b\"],\n});\n\n// after\nawait collection.add({\n  ids: [\"doc1\", \"doc1-dup\", \"doc2\"], // or dedupe first\n  documents: [\"a\", \"a-dup\", \"b\"],\n});\n\n// or overwrite existing rows in one pass\nawait collection.upsert({\n  ids: [\"doc1\", \"doc2\"],\n  documents: [\"a\", \"b\"],\n});","handlingStrategy":"validation","validationCode":"function assertUniqueIds(ids: string[]): void {\n  const seen = new Set<string>();\n  const dupes: string[] = [];\n  for (const id of ids) {\n    if (seen.has(id)) dupes.push(id);\n    seen.add(id);\n  }\n  if (dupes.length > 0) {\n    throw new Error(`Duplicate ids in batch: ${[...new Set(dupes)].join(\", \")}`);\n  }\n}\n\nassertUniqueIds(ids);\nawait collection.add({ ids, documents });","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Generate ids with crypto.randomUUID() unless you control uniqueness at the source.","Run new Set(ids).size === ids.length before every add()/upsert() batch.","In resumable ingestion, persist already-sent ids and filter them out of later batches.","Prefer upsert() when records may already exist, but still send each id once per call."],"tags":["ids","validation","duplicate-ids","add-records"],"backgroundTag":"duplicate-id-rejection","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}