{"record":{"id":"e62d13b48e7d3bca","repo":"chroma-core/chroma","slug":"expected-ids-to-be-unique-but-found-duplicates-of-e62d13","errorCode":null,"errorMessage":"Expected IDs to be unique, but found duplicates of ${duplicates.slice(0, 5).join(\", \")}, ..., ${duplicates.slice(duplicates.length - 5).join(\", \")}","messagePattern":"Expected IDs to be unique, but found duplicates of (.+?), \\.\\.\\., (.+?)","errorType":"validation","errorClass":"ChromaValueError","httpStatus":null,"severity":"error","filePath":"clients/new-js/packages/chromadb/src/utils.ts","lineNumber":222,"sourceCode":"  if (nonStrings.length > 0) {\n    throw new ChromaValueError(\n      `Found non-string IDs at ${nonStrings.join(\", \")}`,\n    );\n  }\n\n  const seen = new Set();\n  const duplicates = ids.filter((id) => {\n    if (seen.has(id)) {\n      return id;\n    }\n    seen.add(id);\n  });\n  let message = \"Expected IDs to be unique, but found duplicates of\";\n  if (duplicates.length > 0 && duplicates.length <= 5) {\n    throw new ChromaValueError(`${message} ${duplicates.join(\", \")}`);\n  }\n  if (duplicates.length > 0) {\n    throw new ChromaValueError(\n      `${message} ${duplicates.slice(0, 5).join(\", \")}, ..., ${duplicates\n        .slice(duplicates.length - 5)\n        .join(\", \")}`,\n    );\n  }\n};\n\nexport const validateSparseVector = (v: unknown): v is SparseVector => {\n  if (typeof v !== \"object\" || v === null) {\n    return false;\n  }\n\n  const candidate = v as Record<string, unknown>;\n  const indices = candidate.indices;\n  const values = candidate.values;\n\n  if (!Array.isArray(indices) || !Array.isArray(values)) {\n    return false;","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/chroma-core/chroma/blob/aecdd12c8a891610db8653630b066b32ceb678b5/clients/new-js/packages/chromadb/src/utils.ts#L204-L240","documentation":"ChromaValueError thrown by validateIDs (utils.ts:222) — the large-batch variant of the duplicate-ids error. When more than 5 duplicate values exist, the message is truncated to the first 5, an ellipsis, and the last 5 duplicates so the error stays readable on big batches.","triggerScenarios":"Same as 278 but at scale: bulk imports of 100k rows whose id field (e.g. URL, SKU) repeats more than 5 times; a retry loop that triple-appended a large chunk; id column built from a low-cardinality key.","commonSituations":"Nightly ETL over sources with repeated natural keys; concat of overlapping snapshots without dedupe; hash function colliding on truncated input.","solutions":["Dedupe with a Map keyed on id, keeping the first (or last) record per id, then rebuild aligned arrays","Switch to deterministic unique ids (uuid v5 of the content, or source PK + row number)","For idempotent re-ingestion, fetch existing ids first and split the batch into updates and inserts"],"exampleFix":"// before\nawait collection.add({ ids, documents }); // 100k rows, ids repeat heavily\n// after\nconst byId = new Map();\nids.forEach((id, i) => { if (!byId.has(id)) byId.set(id, i); });\nconst idx = [...byId.values()];\nawait collection.add({ ids: idx.map(i => ids[i]), documents: idx.map(i => documents[i]) });","handlingStrategy":"validation","validationCode":"function dedupeById(rows) {\n  const m = new Map();\n  for (const r of rows) if (!m.has(r.id)) m.set(r.id, r);\n  return [...m.values()];\n}\nconst rows = dedupeById(rawRows);\nawait collection.add({ ids: rows.map(r => r.id), documents: rows.map(r => r.text) });","typeGuard":"const hasUniqueIds = (ids) => new Set(ids).size === ids.length;","tryCatchPattern":"try { await collection.add({ ids, documents }); } catch (e) { if (e instanceof ChromaValueError && /found duplicates of/.test(e.message)) { /* rebuild from a Map keyed by id, keep first, retry once */ } else throw e; }","preventionTips":["Dedupe large imports with a Map keyed on id before building arrays","Use deterministic unique ids (uuid v5 / source PK + row no.)","Split batches into inserts and updates via a pre-flight get() of existing ids"],"tags":["chromadb","validation","ids","duplicates","large-batch"],"backgroundTag":"duplicate-ids","analyzedSha":"aecdd12c8a891610db8653630b066b32ceb678b5","analyzedAt":"2026-08-16T21:53:27.228Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}