{"record":{"id":"b9c86b514fe4c642","repo":"chroma-core/chroma","slug":"expected-ids-to-be-unique-but-found-duplicates-of","errorCode":null,"errorMessage":"Expected IDs to be unique, but found duplicates of ${duplicates.join(\", \")}","messagePattern":"Expected IDs to be unique, but found duplicates of (.+?)","errorType":"validation","errorClass":"ChromaValueError","httpStatus":null,"severity":"error","filePath":"clients/new-js/packages/chromadb/src/utils.ts","lineNumber":219,"sourceCode":"    .filter(([id, _]) => typeof id !== \"string\")\n    .map(([_, i]) => i);\n\n  if (nonStrings.length > 0) {\n    throw new ChromaValueError(\n      `Found non-string IDs at ${nonStrings.join(\", \")}`,\n    );\n  }\n\n  const seen = new Set();\n  const duplicates = ids.filter((id) => {\n    if (seen.has(id)) {\n      return id;\n    }\n    seen.add(id);\n  });\n  let message = \"Expected IDs to be unique, but found duplicates of\";\n  if (duplicates.length > 0 && duplicates.length <= 5) {\n    throw new ChromaValueError(`${message} ${duplicates.join(\", \")}`);\n  }\n  if (duplicates.length > 0) {\n    throw new ChromaValueError(\n      `${message} ${duplicates.slice(0, 5).join(\", \")}, ..., ${duplicates\n        .slice(duplicates.length - 5)\n        .join(\", \")}`,\n    );\n  }\n};\n\nexport const validateSparseVector = (v: unknown): v is SparseVector => {\n  if (typeof v !== \"object\" || v === null) {\n    return false;\n  }\n\n  const candidate = v as Record<string, unknown>;\n  const indices = candidate.indices;\n  const values = candidate.values;","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/clients/new-js/packages/chromadb/src/utils.ts#L201-L237","documentation":"ChromaValueError thrown by validateIDs (utils.ts:219) when the ids array contains duplicates and there are 5 or fewer distinct duplicate values. Chroma ids are primary keys — re-adding the same id in one batch is ambiguous (insert-or-dedupe behavior is server policy), so the client lists the duplicated values in the message.","triggerScenarios":"collection.add({ ids: ['a','b','a'], documents }) — same id twice in one call; batches assembled from overlapping sources (retry + fresh rows); id generation keyed on a non-unique field like a category or date.","commonSituations":"Retried ingestion re-appending rows already built in this batch; ids derived from filenames that collide on case or truncation; union of two datasets without dedupe.","solutions":["Dedupe before sending: [...new Set(ids)] (aligning the row arrays accordingly)","Generate ids from a stable unique key (content hash, UUID) instead of a business field that can repeat","For true upserts, use collection.update() semantics or check existing ids via get() first"],"exampleFix":"// before\nawait collection.add({ ids, documents }); // ids has 'a' twice\n// after\nconst seen = new Set();\nconst keep = ids.map(id => !seen.has(id) && seen.add(id));\nawait collection.add({ ids: ids.filter((_, i) => keep[i]), documents: documents.filter((_, i) => keep[i]) });","handlingStrategy":"validation","validationCode":"function dedupeRows(ids, ...arrays) {\n  const seen = new Set();\n  const keep = ids.map(id => (seen.has(id) ? false : (seen.add(id), true)));\n  return [ids.filter((_, i) => keep[i]), ...arrays.map(a => a.filter((_, i) => keep[i]))];\n}","typeGuard":"const hasUniqueIds = (ids) => new Set(ids).size === ids.length;","tryCatchPattern":"try { await collection.add({ ids, documents }); } catch (e) { if (e instanceof ChromaValueError && /found duplicates of/.test(e.message)) { [ids, documents] = dedupeRows(ids, documents); await collection.add({ ids, documents }); } else throw e; }","preventionTips":["Dedupe ids (with aligned row arrays) before every add","Derive ids from a stable unique key or content hash","Split overlapping datasets before batching"],"tags":["chromadb","validation","ids","duplicates"],"backgroundTag":"duplicate-ids","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}