What causes "lone high UTF-16 surrogate escape" in gastownhall/beads?

Repair the data so the high surrogate is paired with a valid low surrogate (\uDC00–\uDFFF).

lone high UTF-16 surrogate escape

Error message

lone high UTF-16 surrogate escape

What it means

JSON cannot represent an unpaired high UTF-16 surrogate (U+D800–U+DBFF). When checkJSONSurrogates finds a high surrogate escape not immediately followed by a valid low surrogate escape, it rejects the value with 'lone high UTF-16 surrogate escape' to prevent producing invalid UTF-8 during migration.

Source

Thrown at internal/migration/legacysqlite/reader.go:661

		if i+1 >= len(raw) {
			return fmt.Errorf("truncated JSON escape")
		}
		if raw[i+1] != 'u' {
			i++
			continue
		}
		if i+6 > len(raw) {
			return fmt.Errorf("truncated JSON Unicode escape")
		}
		code, err := strconv.ParseUint(raw[i+2:i+6], 16, 16)
		if err != nil {
			return fmt.Errorf("invalid JSON Unicode escape")
		}
		switch {
		case code >= 0xd800 && code <= 0xdbff:
			next := i + 6
			if next+6 > len(raw) || raw[next] != '\\' || raw[next+1] != 'u' {
				return fmt.Errorf("lone high UTF-16 surrogate escape")
			}
			low, err := strconv.ParseUint(raw[next+2:next+6], 16, 16)
			if err != nil || low < 0xdc00 || low > 0xdfff {
				return fmt.Errorf("lone high UTF-16 surrogate escape")
			}
			i = next + 5
		case code >= 0xdc00 && code <= 0xdfff:
			return fmt.Errorf("lone low UTF-16 surrogate escape")
		default:
			i += 5
		}
	}
	return nil
}

func validateCurrentTextBytes(issue *types.Issue) error {
	if len(issue.Payload) > currentTextBytes {
		return fmt.Errorf("legacy SQLite issue payload is %d bytes (current TEXT maximum %d)", len(issue.Payload), currentTextBytes)

View on GitHub (pinned to 71377f2769)

Solutions

Repair the data so the high surrogate is paired with a valid low surrogate (\uDC00–\uDFFF).
Replace the surrogate pair with the actual character encoded directly in UTF-8.
Strip or replace lone surrogates (e.g. U+FFFD) before migration.

Example fix

// before
"emoji": "\ud83d"
// after
"emoji": "\ud83d\ude00"

Defensive patterns

Strategy: validation

Validate before calling

if regexp.MustCompile(`\\ud[89ab][0-9a-fA-F]{2}`).MatchString(raw) && !hasPairedLowSurrogate(raw) { return errors.New("lone high surrogate") }

Type guard

func hasPairedLowSurrogate(s string) bool { return regexp.MustCompile(`\\ud[89ab][0-9a-fA-F]{2}\\ud[c-f][0-9a-fA-F]{2}`).MatchString(s) }

Try / catch

err := checkJSONSurrogates(raw); if err != nil && strings.Contains(err.Error(), "surrogate") { raw = strings.ToValidUTF8(raw, string(utf8.RuneError)) }

Prevention

Encode astral characters as direct UTF-8, not surrogate pairs
Use encoders that reject lone surrogates (e.g. encoding/json does)
Scan legacy data for D800-DFFF escapes before migration

When it happens

Trigger: A \uD800–\uDBFF escape is either the last escape in the string, not followed by '\\u', or followed by a \uDC00–\uDFFF escape that fails to parse or is out of the low-surrogate range.

Common situations: Legacy data encoded from broken UTF-16 sources, strings split across columns, or encoders that emitted surrogate halves incorrectly.

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/b0dfea7e0ac7d675. Report an issue: GitHub.