gastownhall/beads · error

legacy SQLite %s contains invalid UTF-8

Error message

legacy SQLite %s contains invalid UTF-8

What it means

checkUTF8 verifies that every string field read from the legacy SQLite database is valid UTF-8 before it is used anywhere else in the pipeline. SQLite itself accepts arbitrary bytes in TEXT columns, so the migration explicitly guards against mojibake/binary garbage. The message names the field (e.g. 'title', 'description', a label, or a waiter entry) that failed the check.

Source

Thrown at internal/migration/legacysqlite/reader.go:632

		{"issue target", issue.Target},
		{"issue payload", issue.Payload},
	}
	if issue.ExternalRef != nil {
		fields = append(fields, currentString{"issue external_ref", *issue.ExternalRef})
	}
	if issue.CompactedAtCommit != nil {
		fields = append(fields, currentString{"issue compacted_at_commit", *issue.CompactedAtCommit})
	}
	for i, waiter := range issue.Waiters {
		fields = append(fields, currentString{fmt.Sprintf("issue waiters[%d]", i), waiter})
	}
	return checkUTF8(fields...)
}

func checkUTF8(fields ...currentString) error {
	for _, field := range fields {
		if !utf8.ValidString(field.value) {
			return fmt.Errorf("legacy SQLite %s contains invalid UTF-8", field.name)
		}
	}
	return nil
}

func checkJSONSurrogates(raw string) error {
	for i := 0; i < len(raw); i++ {
		if raw[i] != '\\' {
			continue
		}
		if i+1 >= len(raw) {
			return fmt.Errorf("truncated JSON escape")
		}
		if raw[i+1] != 'u' {
			i++
			continue
		}
		if i+6 > len(raw) {

View on GitHub (pinned to 71377f2769)

Solutions

1. Identify the bad bytes: sqlite3 legacy.db "SELECT hex(title) FROM issues WHERE id='<id>'" (or the named field's table) and check for invalid sequences
2. Re-encode the value from its source encoding to UTF-8, e.g. iconv -f WINDOWS-1252 -t UTF-8, and UPDATE the row
3. If the text is unrecoverable, replace the field with a sanitized placeholder and preserve the original bytes in a backup
4. Re-run the migration

Example fix

// before (Windows-1252 bytes in TEXT column)
title = 0x436166E9   -- 'Café' in Latin-1
// after
UPDATE issues SET title = 'Café' WHERE id = '<id>';  -- valid UTF-8: 0x436166C3A9

def utf8_ok(fields):
    bad = []
    for name, value in fields:
        if isinstance(value, str):
            try:
                value.encode('utf-8')
            except UnicodeEncodeError:
                bad.append(name)
        elif isinstance(value, bytes):
            try:
                value.decode('utf-8')
            except UnicodeDecodeError:
                bad.append(name)
    return bad  # empty list means all fields valid

def is_valid_utf8(v) -> bool:
    if isinstance(v, bytes):
        try:
            v.decode('utf-8')
            return True
        except UnicodeDecodeError:
            return False
    return isinstance(v, str)

When it happens

Trigger: Any legacy string column (issue title/description, labels, comments, waiters, dependency rows) contains bytes that are not valid UTF-8 — e.g. raw Latin-1/Windows-1252 text, truncated multi-byte sequences cut by a substring operation, or BLOB data stored in a TEXT column.

Common situations: Data imported from tools using a different encoding (Windows-1252 notes), comments truncated mid-multibyte-character by length-limited writers, corruption from non-UTF8-aware copy operations.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• legacy SQLite issue %s metadata: %w
• sealed legacy SQLite database does not match source fingerpr
• sealed legacy SQLite WAL does not match source fingerprint
• legacy SQLite source changed while sealing
• legacy SQLite source %q must not be a symlink

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/0866b0539a6f29cf. Report an issue: GitHub.