{"record":{"id":"3a2712c868fdf449","repo":"thedotmack/claude-mem","slug":"skipped-unformattable-row-during-backfill","errorCode":null,"errorMessage":"Skipped unformattable row during backfill","messagePattern":"Skipped unformattable row during backfill","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/services/sync/ChromaSync.ts","lineNumber":831,"sourceCode":"   */\n  private async backfillKind<T extends { id: number }>(\n    rows: T[],\n    formatDocs: (row: T) => ChromaDocument[],\n    kind: 'observations' | 'summaries' | 'prompts',\n    backfillProject: string\n  ): Promise<number> {\n    const rowsWithDocs: Array<{ row: T; docs: ChromaDocument[] }> = [];\n    for (const row of rows) {\n      try {\n        rowsWithDocs.push({ row, docs: formatDocs(row) });\n      } catch (error) {\n        // A single unformattable row (e.g. a malformed JSON column) must not\n        // abort the whole backfill. Skip and log it so the rest of the run\n        // proceeds. Mark the id pending so a later higher-id row that advances\n        // the watermark does not strand it: the pending path re-fetches it on\n        // the next run, which self-heals once its format issue is resolved.\n        ChromaSyncState.markPending(backfillProject, kind, [row.id]);\n        logger.warn('CHROMA_SYNC', 'Skipped unformattable row during backfill', {\n          project: backfillProject,\n          kind,\n          rowId: row.id\n        }, error instanceof Error ? error : new Error(String(error)));\n      }\n    }\n    const totalDocs = rowsWithDocs.reduce((sum, { docs }) => sum + docs.length, 0);\n    let processedDocs = 0;\n    let consecutiveFailures = 0;\n\n    for (let rowIndex = 0; rowIndex < rowsWithDocs.length; rowIndex += 1) {\n      const { row, docs } = rowsWithDocs[rowIndex];\n      if (docs.length === 0) {\n        continue;\n      }\n\n      let rowComplete = true;\n      for (let i = 0; i < docs.length; i += this.BATCH_SIZE) {","sourceCodeStart":813,"sourceCodeEnd":849,"githubUrl":"https://github.com/thedotmack/claude-mem/blob/d8bc9755e74915e5c3b999181e10a67c889bce2a/src/services/sync/ChromaSync.ts#L813-L849","documentation":"During backfillKind, a single row that cannot be formatted into documents (e.g. malformed JSON column) is skipped rather than aborting the whole backfill. The row id is marked pending so a later run re-fetches it, self-healing once the data is fixed.","triggerScenarios":"formatDocument/row mapping throws for one row — malformed JSON in a column, null in a NOT-NULL-expected field, unexpected row shape from an older schema.","commonSituations":"Schema migrations leaving legacy rows, hand-edited databases, partially written rows from a crashed writer.","solutions":["Find the offending rowId in the warning and inspect/fix its column data in SQLite.","Rely on the pending mechanism: fix the data and the next backfill re-syncs it.","Fix the formatting function to tolerate the legacy shape if many rows share it.","Add a data migration to normalize old-format rows."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// pre-validate rows before backfill\nconst bad = rows.filter((r) => !isFormattable(r));\nif (bad.length) console.warn(`backfill will skip rows: ${bad.map((r) => r.id).join(',')}`);","typeGuard":"const isFormattable = (row: Row): boolean =>\n  typeof row.id === 'number' && typeof row.content === 'string' && safeJsonFields(row);","tryCatchPattern":"try { formatRow(row); }\ncatch (e) {\n  ChromaSyncState.markPending(project, kind, [row.id]);\n  logger.warn('skipped unformattable row', { rowId: row.id }, e);\n}","preventionTips":["Run schema migrations to normalize legacy rows","Wrap per-row formatting so one bad row cannot abort the batch","Monitor skipped-row warnings for rising rates","Fix flagged rows promptly so pending retries succeed"],"tags":["backfill","sync","data-format","resilience"],"backgroundTag":"schema-validation-failed","analyzedSha":"d8bc9755e74915e5c3b999181e10a67c889bce2a","analyzedAt":"2026-09-17T16:40:26.182Z","contentChangedAt":"2026-09-17T16:40:26.182Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}