{"record":{"id":"b0dfea7e0ac7d675","repo":"gastownhall/beads","slug":"lone-high-utf-16-surrogate-escape","errorCode":null,"errorMessage":"lone high UTF-16 surrogate escape","messagePattern":"lone high UTF-16 surrogate escape","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/migration/legacysqlite/reader.go","lineNumber":661,"sourceCode":"\t\tif i+1 >= len(raw) {\n\t\t\treturn fmt.Errorf(\"truncated JSON escape\")\n\t\t}\n\t\tif raw[i+1] != 'u' {\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\t\tif i+6 > len(raw) {\n\t\t\treturn fmt.Errorf(\"truncated JSON Unicode escape\")\n\t\t}\n\t\tcode, err := strconv.ParseUint(raw[i+2:i+6], 16, 16)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"invalid JSON Unicode escape\")\n\t\t}\n\t\tswitch {\n\t\tcase code >= 0xd800 && code <= 0xdbff:\n\t\t\tnext := i + 6\n\t\t\tif next+6 > len(raw) || raw[next] != '\\\\' || raw[next+1] != 'u' {\n\t\t\t\treturn fmt.Errorf(\"lone high UTF-16 surrogate escape\")\n\t\t\t}\n\t\t\tlow, err := strconv.ParseUint(raw[next+2:next+6], 16, 16)\n\t\t\tif err != nil || low < 0xdc00 || low > 0xdfff {\n\t\t\t\treturn fmt.Errorf(\"lone high UTF-16 surrogate escape\")\n\t\t\t}\n\t\t\ti = next + 5\n\t\tcase code >= 0xdc00 && code <= 0xdfff:\n\t\t\treturn fmt.Errorf(\"lone low UTF-16 surrogate escape\")\n\t\tdefault:\n\t\t\ti += 5\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc validateCurrentTextBytes(issue *types.Issue) error {\n\tif len(issue.Payload) > currentTextBytes {\n\t\treturn fmt.Errorf(\"legacy SQLite issue payload is %d bytes (current TEXT maximum %d)\", len(issue.Payload), currentTextBytes)","sourceCodeStart":643,"sourceCodeEnd":679,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/migration/legacysqlite/reader.go#L643-L679","documentation":"JSON cannot represent an unpaired high UTF-16 surrogate (U+D800–U+DBFF). When checkJSONSurrogates finds a high surrogate escape not immediately followed by a valid low surrogate escape, it rejects the value with 'lone high UTF-16 surrogate escape' to prevent producing invalid UTF-8 during migration.","triggerScenarios":"A \\uD800–\\uDBFF escape is either the last escape in the string, not followed by '\\\\u', or followed by a \\uDC00–\\uDFFF escape that fails to parse or is out of the low-surrogate range.","commonSituations":"Legacy data encoded from broken UTF-16 sources, strings split across columns, or encoders that emitted surrogate halves incorrectly.","solutions":["Repair the data so the high surrogate is paired with a valid low surrogate (\\uDC00–\\uDFFF).","Replace the surrogate pair with the actual character encoded directly in UTF-8.","Strip or replace lone surrogates (e.g. U+FFFD) before migration."],"exampleFix":"// before\n\"emoji\": \"\\ud83d\"\n// after\n\"emoji\": \"\\ud83d\\ude00\"","handlingStrategy":"validation","validationCode":"if regexp.MustCompile(`\\\\ud[89ab][0-9a-fA-F]{2}`).MatchString(raw) && !hasPairedLowSurrogate(raw) { return errors.New(\"lone high surrogate\") }","typeGuard":"func hasPairedLowSurrogate(s string) bool { return regexp.MustCompile(`\\\\ud[89ab][0-9a-fA-F]{2}\\\\ud[c-f][0-9a-fA-F]{2}`).MatchString(s) }","tryCatchPattern":"err := checkJSONSurrogates(raw); if err != nil && strings.Contains(err.Error(), \"surrogate\") { raw = strings.ToValidUTF8(raw, string(utf8.RuneError)) }","preventionTips":["Encode astral characters as direct UTF-8, not surrogate pairs","Use encoders that reject lone surrogates (e.g. encoding/json does)","Scan legacy data for D800-DFFF escapes before migration"],"tags":["json","utf-16","surrogate","migration"],"backgroundTag":"lone-surrogate-escape","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}