apache/beam · error

non-UTF8 compliant string found

Error message

non-UTF8 compliant string found: %q

What it means

This error is raised by the Beam Go pipeline vet checker when a string field inside a struct element in the pipeline graph is not valid UTF-8. Beam requires all string data in PCollections to be UTF-8 compliant, and checkStructFieldsUTF8 walks struct fields via reflection to catch violations before the pipeline runs.

Solutions

1. Fix the data source so strings are valid UTF-8 (decode with a charset converter such as golang.org/x/text/encoding before inserting into the pipeline)
2. Validate input data with utf8.ValidString before adding elements and handle invalid records explicitly
3. Replace invalid bytes with utf8.RuneError using a repair pass (e.g. strings.ToValidUTF8)
4. Store binary data as []byte instead of string fields

Example fix

// before
s := string(rawLatin1Bytes)
// after
s := strings.ToValidUTF8(string(rawLatin1Bytes), string(utf8.RuneError))

if !utf8.ValidString(s) { return fmt.Errorf("invalid UTF-8 in %q", s) } // run before adding elements to the pipeline

func isValidUTF8(s string) bool { return utf8.ValidString(s) }

When it happens

Trigger: Passing a struct to a beam.Create or similar transform whose string field bytes are invalid UTF-8 (e.g. raw binary data, Latin-1 encoded bytes, or truncated multi-byte sequences) stored in a reflect.String field.

Common situations: Reading legacy-encoded text files (ISO-8859-1/Windows-1252), decoding binary protocols into strings without conversion, or data corruption from byte slicing that splits a multi-byte UTF-8 character.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• empty graph
• invalid pipeline
• strictness check failed
• translation failed
• unexpected edge

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6ab24387a6ba9677. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/vet/vet.go:555

		for i := 0; i < v.Len(); i++ {
			if err := e.checkStructFieldsUTF8(v.Index(i), seen); err != nil {
				return err
			}
		}
	case reflect.Map:
		iter := v.MapRange()
		for iter.Next() {
			if err := e.checkStructFieldsUTF8(iter.Key(), seen); err != nil {
				return err
			}
			if err := e.checkStructFieldsUTF8(iter.Value(), seen); err != nil {
				return err
			}
		}
	case reflect.String:
		str := v.String()
		if !utf8.ValidString(str) {
			return fmt.Errorf("non-UTF8 compliant string found: %q", str)
		}
	}
	return nil
}

// We need to take graph.Fns (which can be created from any from graph.NewFn)
// and convert them to all needed function caller signatures,
// and emitters.
//
// The type assertion shim Funcs need to be registered with reflectx.RegisterFunc
// Emitters need to be registered with exec.RegisterEmitter
// Iterators with exec.RegisterInput
// The types need to be registered with beam.RegisterType
// The user functions need to be registered with beam.RegisterFunction
//
// Registrations are all on the concrete element type, rather than the
// pointer type.

View on GitHub (pinned to 12126d8942)