{"record":{"id":"7563f61caf57ac70","repo":"vitessio/vitess","slug":"cannot-normalize-string-containing-invalid-utf-8","errorCode":null,"errorMessage":"cannot normalize string containing invalid UTF-8: %q","messagePattern":"cannot normalize string containing invalid UTF-8: %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/vtgate/vindexes/unicode.go","lineNumber":46,"sourceCode":"\t\"vitess.io/vitess/go/mysql/collations/vindex/collate\"\n\t\"vitess.io/vitess/go/sqltypes\"\n)\n\n// Shared functions for Unicode string normalization\n// for Vindexes.\n\nfunc unicodeHash(pool *sync.Pool, key sqltypes.Value) ([]byte, error) {\n\tcollator := pool.Get().(*collate.Hasher)\n\tdefer pool.Put(collator)\n\n\tkeyBytes, err := key.ToBytes()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// We cannot pass invalid UTF-8 to the collator.\n\tif !utf8.Valid(keyBytes) {\n\t\treturn nil, fmt.Errorf(\"cannot normalize string containing invalid UTF-8: %q\", keyBytes)\n\t}\n\n\t// Ref: http://dev.mysql.com/doc/refman/5.6/en/char.html.\n\t// Trailing spaces are ignored by MySQL.\n\tkeyBytes = bytes.TrimRight(keyBytes, \" \")\n\n\t// We use the collation key which can be used to\n\t// perform lexical comparisons.\n\treturn collator.Hash(keyBytes), nil\n}\n\nvar collateMD5 = sync.Pool{New: func() any {\n\treturn collate.New(md5.New())\n}}\n\nvar collateXX = sync.Pool{New: func() any {\n\treturn collate.New(XXHashBigEndian{Digest: xxhash.New()})\n}}","sourceCodeStart":28,"sourceCodeEnd":64,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtgate/vindexes/unicode.go#L28-L64","documentation":"The unicode vindex hash (used by unicode and unicode_loose_* vindexes) normalizes input strings with a Unicode collator, which only accepts valid UTF-8. If the key bytes are not valid UTF-8, hashing fails with this error rather than producing wrong or silently lossy mappings.","triggerScenarios":"Hashing a vindex column value containing invalid UTF-8 bytes — typically BLOB/VARBINARY values or non-UTF8 encoded strings (latin1 data) sent to a unicode vindex.","commonSituations":"Application writes legacy latin1-encoded strings into a column backed by a unicode vindex; binary data (packed structs, images) used as a vindex key; connection charset mismatch (client sends cp1251/latin1).","solutions":["Ensure the value is valid UTF-8 before writing — fix the client's character set (SET NAMES utf8mb4) or convert the data.","Convert the column data to UTF-8 (CONVERT(... USING utf8mb4)) and migrate legacy encodings.","If keys are truly binary, use a binary-compatible vindex (binary/binary_md5) instead of a unicode vindex."],"exampleFix":"// before\n\"vindex\": {\"type\": \"unicode_loose_md5\"}  // column holds latin1/binary data\n// after (either fix data)\nUPDATE t SET name = CONVERT(BINARY CONVERT(name USING latin1) USING utf8mb4);\n// or pick a matching vindex\n\"vindex\": {\"type\": \"binary_md5\"}","handlingStrategy":"validation","validationCode":"if !utf8.Valid(keyBytes) {\n    return errors.New(\"vindex key must be valid UTF-8\")\n}","typeGuard":"func isValidUTF8Key(b []byte) bool { return utf8.Valid(b) }","tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"invalid UTF-8\") {\n    // fix client charset or convert the value before hashing\n}","preventionTips":["Force client connections to utf8mb4 charset.","Migrate legacy latin1/binary columns to utf8mb4 before adding unicode vindexes.","Choose binary vindexes for truly binary keys."],"tags":["vindex","unicode","utf-8","encoding"],"backgroundTag":"invalid-utf8-input","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}