{"record":{"id":"0866b0539a6f29cf","repo":"gastownhall/beads","slug":"legacy-sqlite-s-contains-invalid-utf-8","errorCode":null,"errorMessage":"legacy SQLite %s contains invalid UTF-8","messagePattern":"legacy SQLite (.+?) contains invalid UTF-8","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/migration/legacysqlite/reader.go","lineNumber":632,"sourceCode":"\t\t{\"issue target\", issue.Target},\n\t\t{\"issue payload\", issue.Payload},\n\t}\n\tif issue.ExternalRef != nil {\n\t\tfields = append(fields, currentString{\"issue external_ref\", *issue.ExternalRef})\n\t}\n\tif issue.CompactedAtCommit != nil {\n\t\tfields = append(fields, currentString{\"issue compacted_at_commit\", *issue.CompactedAtCommit})\n\t}\n\tfor i, waiter := range issue.Waiters {\n\t\tfields = append(fields, currentString{fmt.Sprintf(\"issue waiters[%d]\", i), waiter})\n\t}\n\treturn checkUTF8(fields...)\n}\n\nfunc checkUTF8(fields ...currentString) error {\n\tfor _, field := range fields {\n\t\tif !utf8.ValidString(field.value) {\n\t\t\treturn fmt.Errorf(\"legacy SQLite %s contains invalid UTF-8\", field.name)\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc checkJSONSurrogates(raw string) error {\n\tfor i := 0; i < len(raw); i++ {\n\t\tif raw[i] != '\\\\' {\n\t\t\tcontinue\n\t\t}\n\t\tif i+1 >= len(raw) {\n\t\t\treturn fmt.Errorf(\"truncated JSON escape\")\n\t\t}\n\t\tif raw[i+1] != 'u' {\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\t\tif i+6 > len(raw) {","sourceCodeStart":614,"sourceCodeEnd":650,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/migration/legacysqlite/reader.go#L614-L650","documentation":"checkUTF8 verifies that every string field read from the legacy SQLite database is valid UTF-8 before it is used anywhere else in the pipeline. SQLite itself accepts arbitrary bytes in TEXT columns, so the migration explicitly guards against mojibake/binary garbage. The message names the field (e.g. 'title', 'description', a label, or a waiter entry) that failed the check.","triggerScenarios":"Any legacy string column (issue title/description, labels, comments, waiters, dependency rows) contains bytes that are not valid UTF-8 — e.g. raw Latin-1/Windows-1252 text, truncated multi-byte sequences cut by a substring operation, or BLOB data stored in a TEXT column.","commonSituations":"Data imported from tools using a different encoding (Windows-1252 notes), comments truncated mid-multibyte-character by length-limited writers, corruption from non-UTF8-aware copy operations.","solutions":["Identify the bad bytes: sqlite3 legacy.db \"SELECT hex(title) FROM issues WHERE id='<id>'\" (or the named field's table) and check for invalid sequences","Re-encode the value from its source encoding to UTF-8, e.g. iconv -f WINDOWS-1252 -t UTF-8, and UPDATE the row","If the text is unrecoverable, replace the field with a sanitized placeholder and preserve the original bytes in a backup","Re-run the migration"],"exampleFix":"// before (Windows-1252 bytes in TEXT column)\ntitle = 0x436166E9   -- 'Café' in Latin-1\n// after\nUPDATE issues SET title = 'Café' WHERE id = '<id>';  -- valid UTF-8: 0x436166C3A9","handlingStrategy":"validation","validationCode":"def utf8_ok(fields):\n    bad = []\n    for name, value in fields:\n        if isinstance(value, str):\n            try:\n                value.encode('utf-8')\n            except UnicodeEncodeError:\n                bad.append(name)\n        elif isinstance(value, bytes):\n            try:\n                value.decode('utf-8')\n            except UnicodeDecodeError:\n                bad.append(name)\n    return bad  # empty list means all fields valid","typeGuard":"def is_valid_utf8(v) -> bool:\n    if isinstance(v, bytes):\n        try:\n            v.decode('utf-8')\n            return True\n        except UnicodeDecodeError:\n            return False\n    return isinstance(v, str)","tryCatchPattern":null,"preventionTips":["Convert the legacy DB to strict UTF-8 (iconv) before migrating","Avoid byte-level truncation of text columns (cut on character boundaries, not bytes)","Open the DB read-only with UTF-8 enforcement and scan all TEXT columns for invalid sequences as pre-flight","Store BLOBs in BLOB columns, never TEXT"],"tags":["migration","sqlite","utf-8","encoding"],"backgroundTag":"invalid-utf8-data","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}