chroma-core/chroma · error · ChromaValueError

Expected IDs to be unique, but found duplicates of ${duplica

Error message

Expected IDs to be unique, but found duplicates of ${duplicates.slice(0, 5).join(", ")}, ..., ${duplicates.slice(duplicates.length - 5).join(", ")}

What it means

ChromaValueError thrown by validateIDs (utils.ts:222) — the large-batch variant of the duplicate-ids error. When more than 5 duplicate values exist, the message is truncated to the first 5, an ellipsis, and the last 5 duplicates so the error stays readable on big batches.

Source

Thrown at clients/new-js/packages/chromadb/src/utils.ts:222

  if (nonStrings.length > 0) {
    throw new ChromaValueError(
      `Found non-string IDs at ${nonStrings.join(", ")}`,
    );
  }

  const seen = new Set();
  const duplicates = ids.filter((id) => {
    if (seen.has(id)) {
      return id;
    }
    seen.add(id);
  });
  let message = "Expected IDs to be unique, but found duplicates of";
  if (duplicates.length > 0 && duplicates.length <= 5) {
    throw new ChromaValueError(`${message} ${duplicates.join(", ")}`);
  }
  if (duplicates.length > 0) {
    throw new ChromaValueError(
      `${message} ${duplicates.slice(0, 5).join(", ")}, ..., ${duplicates
        .slice(duplicates.length - 5)
        .join(", ")}`,
    );
  }
};

export const validateSparseVector = (v: unknown): v is SparseVector => {
  if (typeof v !== "object" || v === null) {
    return false;
  }

  const candidate = v as Record<string, unknown>;
  const indices = candidate.indices;
  const values = candidate.values;

  if (!Array.isArray(indices) || !Array.isArray(values)) {
    return false;

View on GitHub (pinned to aecdd12c8a)

Solutions

1. Dedupe with a Map keyed on id, keeping the first (or last) record per id, then rebuild aligned arrays
2. Switch to deterministic unique ids (uuid v5 of the content, or source PK + row number)
3. For idempotent re-ingestion, fetch existing ids first and split the batch into updates and inserts

Example fix

// before
await collection.add({ ids, documents }); // 100k rows, ids repeat heavily
// after
const byId = new Map();
ids.forEach((id, i) => { if (!byId.has(id)) byId.set(id, i); });
const idx = [...byId.values()];
await collection.add({ ids: idx.map(i => ids[i]), documents: idx.map(i => documents[i]) });

function dedupeById(rows) {
  const m = new Map();
  for (const r of rows) if (!m.has(r.id)) m.set(r.id, r);
  return [...m.values()];
}
const rows = dedupeById(rawRows);
await collection.add({ ids: rows.map(r => r.id), documents: rows.map(r => r.text) });

const hasUniqueIds = (ids) => new Set(ids).size === ids.length;

try { await collection.add({ ids, documents }); } catch (e) { if (e instanceof ChromaValueError && /found duplicates of/.test(e.message)) { /* rebuild from a Map keyed by id, keep first, retry once */ } else throw e; }

When it happens

Trigger: Same as 278 but at scale: bulk imports of 100k rows whose id field (e.g. URL, SKU) repeats more than 5 times; a retry loop that triple-appended a large chunk; id column built from a low-cardinality key.

Common situations: Nightly ETL over sources with repeated natural keys; concat of overlapping snapshots without dedupe; hash function colliding on truncated input.

Related errors

• Expected IDs to be unique, but found duplicates of ${duplica
• Expected 'ids' to be an array, but got ${typeof ids}
• Expected 'ids' to be a non-empty list
• Found non-string IDs at ${nonStrings.join(", ")}
• Expected IDs to be a list, got {type(ids).__name__} as IDs

AI-assisted analysis of chroma-core/chroma@aecdd12c8a (2026-08-16). Data as JSON: /api/errors/e62d13b48e7d3bca. Report an issue: GitHub.