can1357/oh-my-pi · error · Error

import_all: duplicate id ${item.id} in the imported batch. D

Error message

import_all: duplicate id ${item.id} in the imported batch. Deduplicate the input before calling.

What it means

importAll rejects batches containing two triples with the same explicit id, throwing before starting its transaction. The check exists because duplicate ids would violate the importer's id-preserving semantics and fail mid-transaction; it requires the caller to deduplicate first.

Source

Thrown at packages/mnemopi/src/core/triples.ts:338

			.query("SELECT DISTINCT object FROM triples WHERE predicate = ? ORDER BY object")
			.all(predicate)
			.map(row => (row as { object: string }).object);
	}
	exportAll(): TripleRow[] {
		return this.conn.query(`SELECT ${TRIPLE_COLUMNS} FROM triples ORDER BY id`).all().map(rowToTriple);
	}
	importAll(triples: readonly TripleImportRow[], force = false): TripleImportStats {
		const stats: TripleImportStats = {
			inserted: 0,
			skipped: 0,
			overwritten: 0,
			imported_renumbered: 0,
		};
		const seen = new Set<number>();
		for (const item of triples) {
			if (item.id === undefined || item.id === null) continue;
			if (seen.has(item.id))
				throw new Error(
					`import_all: duplicate id ${item.id} in the imported batch. Deduplicate the input before calling.`,
				);
			seen.add(item.id);
		}

		this.conn.run("BEGIN IMMEDIATE");
		try {
			const existing = new Map<number, ContentSnapshot>();
			for (const row of this.conn.query(`SELECT ${TRIPLE_COLUMNS} FROM triples`).all().map(rowToTriple)) {
				existing.set(row.id, contentFromRow(row));
			}
			const explicitNoCollision: TripleImportRow[] = [];
			const noId: TripleImportRow[] = [];
			const collisions: TripleImportRow[] = [];
			for (const item of triples) {
				const id = item.id;
				if (id === undefined || id === null) noId.push(item);
				else if (existing.has(id)) collisions.push(item);

View on GitHub (pinned to 9690622007)

Solutions

Deduplicate the input array by id before calling importAll (e.g. new Map(triples.map(t => [t.id, t])).values())
If ids should be reassigned, strip the id field (undefined) so the importer generates fresh ids
Split the batch so each id appears only once per call

Example fix

// before
await store.importAll([...oldBatch, ...reExported]);
// after
const deduped = [...new Map([...oldBatch, ...reExported].map(t => [t.id, t])).values()];
await store.importAll(deduped);

Defensive patterns

Strategy: validation

Validate before calling

function dedupeById(triples) {
	const map = new Map();
	for (const t of triples) {
		if (t.id === undefined || t.id === null) { map.set(Symbol(), t); continue; }
		if (map.has(t.id)) throw new Error(`duplicate id ${t.id} in import batch`);
		map.set(t.id, t);
	}
	return [...map.values()];
}
await store.importAll(dedupeById(triples));

Try / catch

try {
	await store.importAll(batch);
} catch (err) {
	if (err.message.includes("duplicate id")) {
		const id = Number(/duplicate id (\d+)/.exec(err.message)?.[1]);
		batch = batch.filter((t, i, a) => a.findIndex(x => x.id === t.id) === i);
		await store.importAll(batch);
	} else throw err;
}

Prevention

Deduplicate by id before every importAll call
Don't concatenate overlapping exports into one batch
Assign fresh ids (strip id) when re-importing previously imported data
Idempotency-check imports against already-stored ids

When it happens

Trigger: Calling importAll(triples) where two or more items have the same numeric id (and id is not undefined/null); merging two export files that overlap; re-importing a batch that already contains previously imported ids concatenated together.

Common situations: Combining exports from two databases with overlapping id ranges; appending a re-export to an existing batch; upstream data pipeline emitting duplicate rows; retry logic that re-appends already-batched items.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/7fc7acb60c073ba8. Report an issue: GitHub.