{"record":{"id":"6ab24387a6ba9677","repo":"apache/beam","slug":"non-utf8-compliant-string-found-q","errorCode":null,"errorMessage":"non-UTF8 compliant string found: %q","messagePattern":"non-UTF8 compliant string found: %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/runners/vet/vet.go","lineNumber":555,"sourceCode":"\t\tfor i := 0; i < v.Len(); i++ {\n\t\t\tif err := e.checkStructFieldsUTF8(v.Index(i), seen); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\tcase reflect.Map:\n\t\titer := v.MapRange()\n\t\tfor iter.Next() {\n\t\t\tif err := e.checkStructFieldsUTF8(iter.Key(), seen); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif err := e.checkStructFieldsUTF8(iter.Value(), seen); err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\tcase reflect.String:\n\t\tstr := v.String()\n\t\tif !utf8.ValidString(str) {\n\t\t\treturn fmt.Errorf(\"non-UTF8 compliant string found: %q\", str)\n\t\t}\n\t}\n\treturn nil\n}\n\n// We need to take graph.Fns (which can be created from any from graph.NewFn)\n// and convert them to all needed function caller signatures,\n// and emitters.\n//\n// The type assertion shim Funcs need to be registered with reflectx.RegisterFunc\n// Emitters need to be registered with exec.RegisterEmitter\n// Iterators with exec.RegisterInput\n// The types need to be registered with beam.RegisterType\n// The user functions need to be registered with beam.RegisterFunction\n//\n// Registrations are all on the concrete element type, rather than the\n// pointer type.\n","sourceCodeStart":537,"sourceCodeEnd":573,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/runners/vet/vet.go#L537-L573","documentation":"This error is raised by the Beam Go pipeline vet checker when a string field inside a struct element in the pipeline graph is not valid UTF-8. Beam requires all string data in PCollections to be UTF-8 compliant, and checkStructFieldsUTF8 walks struct fields via reflection to catch violations before the pipeline runs.","triggerScenarios":"Passing a struct to a beam.Create or similar transform whose string field bytes are invalid UTF-8 (e.g. raw binary data, Latin-1 encoded bytes, or truncated multi-byte sequences) stored in a reflect.String field.","commonSituations":"Reading legacy-encoded text files (ISO-8859-1/Windows-1252), decoding binary protocols into strings without conversion, or data corruption from byte slicing that splits a multi-byte UTF-8 character.","solutions":["Fix the data source so strings are valid UTF-8 (decode with a charset converter such as golang.org/x/text/encoding before inserting into the pipeline)","Validate input data with utf8.ValidString before adding elements and handle invalid records explicitly","Replace invalid bytes with utf8.RuneError using a repair pass (e.g. strings.ToValidUTF8)","Store binary data as []byte instead of string fields"],"exampleFix":"// before\ns := string(rawLatin1Bytes)\n// after\ns := strings.ToValidUTF8(string(rawLatin1Bytes), string(utf8.RuneError))","handlingStrategy":"validation","validationCode":"if !utf8.ValidString(s) { return fmt.Errorf(\"invalid UTF-8 in %q\", s) } // run before adding elements to the pipeline","typeGuard":"func isValidUTF8(s string) bool { return utf8.ValidString(s) }","tryCatchPattern":null,"preventionTips":["Decode legacy encodings (Latin-1 etc.) to UTF-8 at ingestion","Run utf8.ValidString checks on external data before pipeline insertion","Use strings.ToValidUTF8 as a sanitization pass","Prefer []byte for binary data instead of string"],"tags":["go","utf8","data-validation","pipeline"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}