apache/beam · error

bad struct encoding

Error message

bad struct encoding

What it means

After decoding the structural DoFn's type, decodeFn unmarshals the JSON Opt payload into a new value of that type with jsonx.UnmarshalFrom; failure is wrapped as "bad struct encoding". The serialized DoFn state cannot be applied to the reconstructed struct.

Solutions

1. Keep the DoFn struct's JSON-visible shape stable between pipeline submission and execution.
2. Check the wrapped jsonx error for the offending field and fix the mismatch (name or type).
3. Add `json:"-"` to fields not meant to travel, and reinitialize them in Setup().
4. Re-submit the pipeline after any change to DoFn struct fields.

Example fix

// before
type myFn struct{ Count int64 } // was Count int before rename on one side

// after
type myFn struct {
    Count int64 `json:"count"` // stable JSON key across versions
}

if err := json.Unmarshal([]byte(optJSON), reflect.New(reflect.TypeOf(myFn{})).Interface()); err != nil {
    return fmt.Errorf("DoFn state JSON incompatible with current struct: %w", err)
}

if err := jsonx.UnmarshalFrom(elem.Interface(), strings.NewReader(u.Opt)); err != nil {
    return fmt.Errorf("structural DoFn state mismatch; resubmit the pipeline with the current struct definition: %w", err)
}

When it happens

Trigger: DecodeMultiEdge decoding a structural DoFn whose u.Opt JSON no longer matches the struct's fields/types in the worker's version of the type.

Common situations: DoFn struct fields renamed, retyped, or removed after the pipeline was serialized; custom UnmarshalJSON methods failing; JSON-incompatible fields added to the struct.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• bad element
• bad symbol
• bad type
• could not unmarshal CoderRef from
• Could not unmarshal SourceConfig

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/998af3e1eb0c9d42. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/serialize.go:310

			wrapped := errors.Wrap(err, "failed to decode userfn")
			return nil, errors.WithContextf(wrapped, "decoding DoFn %v", u)
		}
		fx, err := funcx.New(reflectx.MakeFunc(fn))
		if err != nil {
			wrapped := errors.Wrap(err, "failed to construct userfn")
			return nil, errors.WithContextf(wrapped, "decoding DoFn %v", u)
		}
		return &graph.Fn{Fn: fx}, nil
	}

	t, err := decodeType(u.Type)
	if err != nil {
		wrapped := errors.Wrap(err, "bad type")
		return nil, errors.WithContextf(wrapped, "decoding structural DoFn %v", u)
	}
	elem := reflect.New(t)
	if err := jsonx.UnmarshalFrom(elem.Interface(), strings.NewReader(u.Opt)); err != nil {
		wrapped := errors.Wrap(err, "bad struct encoding")
		return nil, errors.WithContextf(wrapped, "decoding structural DoFn %v", u)
	}
	fn := elem.Elem().Interface()
	return graph.NewFn(fn)
}

// encodeUserFn translates the preprocessed representation of a Beam user function
// into the wire representation, capturing all the inputs and outputs needed.
func encodeUserFn(u *funcx.Fn) (*v1pb.UserFn, error) {
	// TODO(herohde) 5/23/2017: reject closures and dynamic functions. They can't
	// be serialized.

	symbol := u.Fn.Name()
	t, err := encodeType(u.Fn.Type())
	if err != nil {
		wrapped := errors.Wrap(err, "bad function type")
		return nil, errors.WithContextf(wrapped, "encoding userfn %v", u)
	}

View on GitHub (pinned to 12126d8942)