{"record":{"id":"f74f75d311c8b6ea","repo":"mastra-ai/mastra","slug":"dataset-item-identity-history-is-corrupt-for-exter-f74f75","errorCode":null,"errorMessage":"Dataset item identity history is corrupt for externalId: ${first.externalId}","messagePattern":"Dataset item identity history is corrupt for externalId: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"packages/core/src/storage/domains/datasets/inmemory.ts","lineNumber":560,"sourceCode":"    };\n  }\n\n  // --- Bulk operations (SCD-2 internally) ---\n\n  protected async _doBatchInsertItems(input: BatchInsertItemsInput): Promise<DatasetItem[]> {\n    const dataset = this.db.datasets.get(input.datasetId);\n    if (!dataset) {\n      throw new Error(`Dataset not found: ${input.datasetId}`);\n    }\n    if (input.items.length === 0) return [];\n\n    const acceptedByExternalId = new Map<string, { first: DatasetItemRow; current: DatasetItemRow | null }>();\n    for (const rows of this.db.datasetItems.values()) {\n      const first = rows[0];\n      if (!first || first.datasetId !== input.datasetId || !first.externalId) continue;\n      const existing = acceptedByExternalId.get(first.externalId);\n      if (existing && existing.first.id !== first.id) {\n        throw new Error(`Dataset item identity history is corrupt for externalId: ${first.externalId}`);\n      }\n      acceptedByExternalId.set(first.externalId, {\n        first,\n        current: rows.find(row => row.validTo === null && !row.isDeleted) ?? null,\n      });\n    }\n\n    const conflicts = [];\n    const planned = new Map<string, { id: string; item: (typeof input.items)[number] }>();\n    const plannedByExternalId = new Map<string, { id: string; item: (typeof input.items)[number] }>();\n    const resolvedIds: string[] = [];\n\n    for (const [index, item] of input.items.entries()) {\n      if (!item.externalId) {\n        const id = crypto.randomUUID();\n        planned.set(id, { id, item });\n        resolvedIds.push(id);\n        continue;","sourceCodeStart":542,"sourceCodeEnd":578,"githubUrl":"https://github.com/mastra-ai/mastra/blob/75dd419e613fe9c39f846ffc500716141b74fda6/packages/core/src/storage/domains/datasets/inmemory.ts#L542-L578","documentation":"A consistency invariant check inside batchInsertItems: two different item ids were found with the same first (oldest) SCD-2 history row for one externalId. The internal identity history is corrupted and the store refuses to guess which lineage to extend.","triggerScenarios":"Manual mutation of this.db.datasetItems (internal map) by test code; a prior bug or out-of-band write created duplicate history lineages for the same externalId; restoring/partially overwriting the in-memory db from a snapshot.","commonSituations":"Custom tests poking at private state; monkey-patched or subclassed stores that bypass insert paths; corrupted snapshots restored into a fresh in-memory instance.","solutions":["Do not mutate the store's internal db directly; only use public insert/update/delete APIs.","Rebuild the store (fresh instance) and re-insert items so a single lineage per externalId exists.","Find the code path that wrote duplicate lineages for the externalId and fix it.","Ensure externalIds are unique per dataset before batch inserting."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Ensure externalIds are unique per dataset before batch insert\nconst seen = new Set<string>();\nfor (const item of items) {\n  if (item.externalId && seen.has(item.externalId)) {\n    throw new Error(`Duplicate externalId in batch: ${item.externalId}`);\n  }\n  seen.add(item.externalId);\n}","typeGuard":"null","tryCatchPattern":"try {\n  await storage.datasets.batchInsertItems({ datasetId, items });\n} catch (e) {\n  if (String(e).includes('identity history is corrupt')) {\n    // rebuild store: recreate dataset and re-insert items\n  } else throw e;\n}","preventionTips":["Never mutate the store's internal db maps directly.","Only mutate datasets through public APIs.","Keep externalId unique per dataset in your ingestion pipeline."],"tags":["storage","datasets","inmemory","corruption","invariant"],"backgroundTag":"data-corruption-detected","analyzedSha":"75dd419e613fe9c39f846ffc500716141b74fda6","analyzedAt":"2026-08-30T00:15:31.844Z","schemaVersion":2},"datasetVersion":"2026-08-30T03:17:51.788Z"}