vitessio/vitess · error

cannot normalize string containing invalid UTF-8: %q

Error message

cannot normalize string containing invalid UTF-8: %q

What it means

The unicode vindex hash (used by unicode and unicode_loose_* vindexes) normalizes input strings with a Unicode collator, which only accepts valid UTF-8. If the key bytes are not valid UTF-8, hashing fails with this error rather than producing wrong or silently lossy mappings.

Source

Thrown at go/vt/vtgate/vindexes/unicode.go:46

	"vitess.io/vitess/go/mysql/collations/vindex/collate"
	"vitess.io/vitess/go/sqltypes"
)

// Shared functions for Unicode string normalization
// for Vindexes.

func unicodeHash(pool *sync.Pool, key sqltypes.Value) ([]byte, error) {
	collator := pool.Get().(*collate.Hasher)
	defer pool.Put(collator)

	keyBytes, err := key.ToBytes()
	if err != nil {
		return nil, err
	}

	// We cannot pass invalid UTF-8 to the collator.
	if !utf8.Valid(keyBytes) {
		return nil, fmt.Errorf("cannot normalize string containing invalid UTF-8: %q", keyBytes)
	}

	// Ref: http://dev.mysql.com/doc/refman/5.6/en/char.html.
	// Trailing spaces are ignored by MySQL.
	keyBytes = bytes.TrimRight(keyBytes, " ")

	// We use the collation key which can be used to
	// perform lexical comparisons.
	return collator.Hash(keyBytes), nil
}

var collateMD5 = sync.Pool{New: func() any {
	return collate.New(md5.New())
}}

var collateXX = sync.Pool{New: func() any {
	return collate.New(XXHashBigEndian{Digest: xxhash.New()})
}}

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Ensure the value is valid UTF-8 before writing — fix the client's character set (SET NAMES utf8mb4) or convert the data.
2. Convert the column data to UTF-8 (CONVERT(... USING utf8mb4)) and migrate legacy encodings.
3. If keys are truly binary, use a binary-compatible vindex (binary/binary_md5) instead of a unicode vindex.

Example fix

// before
"vindex": {"type": "unicode_loose_md5"}  // column holds latin1/binary data
// after (either fix data)
UPDATE t SET name = CONVERT(BINARY CONVERT(name USING latin1) USING utf8mb4);
// or pick a matching vindex
"vindex": {"type": "binary_md5"}

if !utf8.Valid(keyBytes) {
    return errors.New("vindex key must be valid UTF-8")
}

func isValidUTF8Key(b []byte) bool { return utf8.Valid(b) }

if err != nil && strings.Contains(err.Error(), "invalid UTF-8") {
    // fix client charset or convert the value before hashing
}

When it happens

Trigger: Hashing a vindex column value containing invalid UTF-8 bytes — typically BLOB/VARBINARY values or non-UTF8 encoded strings (latin1 data) sent to a unicode vindex.

Common situations: Application writes legacy latin1-encoded strings into a column backed by a unicode vindex; binary data (packed structs, images) used as a vindex key; connection charset mismatch (client sends cp1251/latin1).

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• UnicodeLooseMD5.Verify: %v
• UnicodeLooseMD5.Map: %v
• UnicodeLooseXXHash.Verify: %v
• UnicodeLooseXXHash.Map: %v
• one or two tables must be specified

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/7563f61caf57ac70. Report an issue: GitHub.