{"record":{"id":"07794d9c44393add","repo":"gastownhall/beads","slug":"s-w-07794d","errorCode":null,"errorMessage":"%s: %w","messagePattern":"%s: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"internal/storage/schema/aux_row_id_backfill.go","lineNumber":255,"sourceCode":"\t// shipped version, but its sentinel proves the rewrite never finished\n\t// (bd-578h9.16).\n\tif mainVersionBefore >= pass.shippedMainVersion && !resume {\n\t\treturn false, nil\n\t}\n\n\t// Sentinel before the first UPDATE: a crash anywhere in the rewrite\n\t// leaves it set, so the next pass resumes (the rewrite is idempotent)\n\t// instead of recording the marker over partially re-keyed rows.\n\tif err := setAuxRekeyInProgress(ctx, db, pass.sentinelKey); err != nil {\n\t\treturn false, fmt.Errorf(\"recording aux rekey sentinel: %w\", err)\n\t}\n\n\twrote := false\n\tfor _, t := range auxRekeyTables {\n\t\tw, err := rekeyAuxRowTable(ctx, db, t)\n\t\twrote = wrote || w\n\t\tif err != nil {\n\t\t\treturn wrote, fmt.Errorf(\"%s: %w\", t.name, err)\n\t\t}\n\t}\n\tif err := clearAuxRekeyInProgress(ctx, db, pass.sentinelKey); err != nil {\n\t\treturn wrote, fmt.Errorf(\"clearing aux rekey sentinel: %w\", err)\n\t}\n\treturn wrote, nil\n}\n\n// rekeyAuxRowTable re-derives the ids of one table. The whole table is grouped\n// by content digest; each digest's rows take the deterministic ids for\n// ordinals 0..n-1. A row already holding one of its group's target ids keeps\n// it (idempotence: re-running never swaps ids within a group), and the\n// remaining rows take the remaining targets in sorted-current-id order. Across\n// clones that assignment may permute within a group of exact-duplicate rows,\n// but duplicates are interchangeable and the id set is identical, so the\n// merged result still converges.\nfunc rekeyAuxRowTable(ctx context.Context, db DBConn, t auxRekeyTable) (bool, error) {\n\t// Skip cleanly if the table or its id column isn't present (older or partial","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/storage/schema/aux_row_id_backfill.go#L237-L273","documentation":"During the rekey rewrite, rekeyAuxRowIDsPending rewrites each of the four aux tables (events, comments, issue_snapshots, compaction_snapshots) via rekeyAuxRowTable. If a table's rewrite fails, this error prefixes the table name to the wrapped cause (`<table>: <cause>`), so the operator knows exactly which table's id re-derivation failed. The sentinel stays set, so the next MigrateUp resumes and the rewrite is idempotent — rows already holding derived ids keep them.","triggerScenarios":"rekeyAuxRowTable failing for one aux table — typically a failed UPDATE/SELECT on that table: lock timeout, duplicate derived id colliding with an existing row, constraint violation, connection drop mid-rewrite, or corrupt row content that breaks the digest derivation.","commonSituations":"Large aux tables hitting lock/timeout limits during migration; a clone with divergent rows producing an id collision; interrupted earlier run leaving a table half-rekeyed (resume path hits an unexpected state); storage engine errors under disk pressure.","solutions":["Read the table-name prefix and the wrapped driver error to identify the failing table and root cause.","Simply re-run MigrateUp: the sentinel guarantees resume and the rewrite is idempotent, so partial progress is preserved.","For duplicate-key errors, inspect rows whose derived id collides (rowid derivation in internal/storage/rowid) and resolve duplicate content rows.","For lock/timeout errors, run the migration during a quiet window or raise lock timeouts, then re-run.","If a table is corrupt, restore it from Dolt history or re-clone before resuming."],"exampleFix":"// before\n// rekey fails with: events: Error 1062: Duplicate entry '<id>' for key 'PRIMARY'\n\n// after\n-- resolve duplicate content rows in the named table, then:\n$ bd migrate   # or just restart bd; the sentinel makes the pass resume idempotently","handlingStrategy":"retry","validationCode":"// pre-check for duplicate-prone divergent rows before migrating:\nfor _, t := range []string{\"events\", \"comments\", \"issue_snapshots\", \"compaction_snapshots\"} {\n    var dupes int\n    db.QueryRow(fmt.Sprintf(`SELECT COUNT(*) FROM (SELECT 1 FROM %s GROUP BY issue_id HAVING COUNT(*) > 100) x`, t)).Scan(&dupes)\n    _ = dupes // inspect pathological groups before running the rewrite\n}","typeGuard":null,"tryCatchPattern":"if err := migrateUp(ctx, db); err != nil {\n    var table, cause string\n    if i := strings.Index(err.Error(), \": \"); i > 0 {\n        table, cause = err.Error()[:i], err.Error()[i+2:]\n    }\n    if strings.Contains(cause, \"Duplicate entry\") {\n        return fmt.Errorf(\"rekey id collision on %s — resolve duplicate content rows, then re-run (resumes idempotently): %w\", table, err)\n    }\n    if strings.Contains(cause, \"Lock wait timeout\") {\n        return retryWithBackoff(func() error { return migrateUp(ctx, db) })\n    }\n    return err\n}","preventionTips":["Re-run MigrateUp after any failure: the sentinel makes the pass resume and the rewrite is idempotent.","Run the migration in a quiet window; large aux tables are sensitive to lock timeouts.","Resolve duplicate content rows before migrating lineages that migrated independently with randomized ids (pre-0037 clones).","Monitor disk space during the backfill — four full-table rewrites are issued.","Parse the table-name prefix in the error to target diagnosis instead of re-deriving it."],"tags":["database","migration","dolt","backfill","crash-recovery"],"backgroundTag":"backfill-table-rewrite-failed","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}