{"record":{"id":"7fc7acb60c073ba8","repo":"can1357/oh-my-pi","slug":"import-all-duplicate-id-item-id-in-the-importe","errorCode":null,"errorMessage":"import_all: duplicate id ${item.id} in the imported batch. Deduplicate the input before calling.","messagePattern":"import_all: duplicate id (.+?) in the imported batch\\. Deduplicate the input before calling\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/mnemopi/src/core/triples.ts","lineNumber":338,"sourceCode":"\t\t\t.query(\"SELECT DISTINCT object FROM triples WHERE predicate = ? ORDER BY object\")\n\t\t\t.all(predicate)\n\t\t\t.map(row => (row as { object: string }).object);\n\t}\n\texportAll(): TripleRow[] {\n\t\treturn this.conn.query(`SELECT ${TRIPLE_COLUMNS} FROM triples ORDER BY id`).all().map(rowToTriple);\n\t}\n\timportAll(triples: readonly TripleImportRow[], force = false): TripleImportStats {\n\t\tconst stats: TripleImportStats = {\n\t\t\tinserted: 0,\n\t\t\tskipped: 0,\n\t\t\toverwritten: 0,\n\t\t\timported_renumbered: 0,\n\t\t};\n\t\tconst seen = new Set<number>();\n\t\tfor (const item of triples) {\n\t\t\tif (item.id === undefined || item.id === null) continue;\n\t\t\tif (seen.has(item.id))\n\t\t\t\tthrow new Error(\n\t\t\t\t\t`import_all: duplicate id ${item.id} in the imported batch. Deduplicate the input before calling.`,\n\t\t\t\t);\n\t\t\tseen.add(item.id);\n\t\t}\n\n\t\tthis.conn.run(\"BEGIN IMMEDIATE\");\n\t\ttry {\n\t\t\tconst existing = new Map<number, ContentSnapshot>();\n\t\t\tfor (const row of this.conn.query(`SELECT ${TRIPLE_COLUMNS} FROM triples`).all().map(rowToTriple)) {\n\t\t\t\texisting.set(row.id, contentFromRow(row));\n\t\t\t}\n\t\t\tconst explicitNoCollision: TripleImportRow[] = [];\n\t\t\tconst noId: TripleImportRow[] = [];\n\t\t\tconst collisions: TripleImportRow[] = [];\n\t\t\tfor (const item of triples) {\n\t\t\t\tconst id = item.id;\n\t\t\t\tif (id === undefined || id === null) noId.push(item);\n\t\t\t\telse if (existing.has(id)) collisions.push(item);","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/mnemopi/src/core/triples.ts#L320-L356","documentation":"importAll rejects batches containing two triples with the same explicit id, throwing before starting its transaction. The check exists because duplicate ids would violate the importer's id-preserving semantics and fail mid-transaction; it requires the caller to deduplicate first.","triggerScenarios":"Calling importAll(triples) where two or more items have the same numeric id (and id is not undefined/null); merging two export files that overlap; re-importing a batch that already contains previously imported ids concatenated together.","commonSituations":"Combining exports from two databases with overlapping id ranges; appending a re-export to an existing batch; upstream data pipeline emitting duplicate rows; retry logic that re-appends already-batched items.","solutions":["Deduplicate the input array by id before calling importAll (e.g. new Map(triples.map(t => [t.id, t])).values())","If ids should be reassigned, strip the id field (undefined) so the importer generates fresh ids","Split the batch so each id appears only once per call"],"exampleFix":"// before\nawait store.importAll([...oldBatch, ...reExported]);\n// after\nconst deduped = [...new Map([...oldBatch, ...reExported].map(t => [t.id, t])).values()];\nawait store.importAll(deduped);","handlingStrategy":"validation","validationCode":"function dedupeById(triples) {\n\tconst map = new Map();\n\tfor (const t of triples) {\n\t\tif (t.id === undefined || t.id === null) { map.set(Symbol(), t); continue; }\n\t\tif (map.has(t.id)) throw new Error(`duplicate id ${t.id} in import batch`);\n\t\tmap.set(t.id, t);\n\t}\n\treturn [...map.values()];\n}\nawait store.importAll(dedupeById(triples));","typeGuard":null,"tryCatchPattern":"try {\n\tawait store.importAll(batch);\n} catch (err) {\n\tif (err.message.includes(\"duplicate id\")) {\n\t\tconst id = Number(/duplicate id (\\d+)/.exec(err.message)?.[1]);\n\t\tbatch = batch.filter((t, i, a) => a.findIndex(x => x.id === t.id) === i);\n\t\tawait store.importAll(batch);\n\t} else throw err;\n}","preventionTips":["Deduplicate by id before every importAll call","Don't concatenate overlapping exports into one batch","Assign fresh ids (strip id) when re-importing previously imported data","Idempotency-check imports against already-stored ids"],"tags":["validation","duplicate","batch-import"],"backgroundTag":"duplicate-key-in-batch","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}