apache/beam · error

encoding partition function

Error message

encoding partition function

What it means

beam.Partition serializes its partition function (via EncodedFunc) into JSON to ship it to the runner. If json.Marshal of the partitionData (including the encoded function) fails, it panics with errors wrapped in the context "encoding partition function". This indicates a marshaling failure of the function's type/encoding metadata.

Source

Thrown at sdks/go/pkg/beam/partition.go:87

		in = []reflect.Type{EventTimeType, k, v}
	} else {
		t := col.Type().Type()
		funcx.MustSatisfy(fn, funcx.Replace(sig, TType, t))
		emit = reflect.FuncOf([]reflect.Type{EventTimeType, t}, nil, false)
		in = []reflect.Type{EventTimeType, t}
	}

	// The partitionFn is a DoFn with a signature that is dependent on the input, so
	// neither reflection nor type-specialization is adequate. Instead, it uses a
	// dynamic function.
	for i := 0; i < n; i++ {
		in = append(in, emit)
	}
	fnT := reflect.FuncOf(in, []reflect.Type{reflectx.Error}, false)

	data, err := json.Marshal(partitionData{KV: typex.IsKV(col.Type()), N: n, Fn: EncodedFunc{Fn: reflectx.MakeFunc(fn)}})
	if err != nil {
		panic(errors.WithContext(err, "encoding partition function"))
	}

	return ParDoN(s, &graph.DynFn{Name: "beam.partitionFn", Data: data, T: fnT, Gen: makePartitionFn}, col)
}

// partitionData contains the data needed for the partition DoFn generator.
type partitionData struct {
	KV bool        `json:"kv"`
	N  int         `json:"n"`
	Fn EncodedFunc `json:"fn"`
}

// partitionFn is a Func with the following underlying type:
//
//	fn : (EventTime, T, emit_1, emit_2, ..., emit_N) -> error
//
// where emit_i : (EventTime, T) -> () and N is given by the encoded
// partitionData value. For any input element, it invokes to the

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure fn is a valid, non-nil Go function with an acceptable partition signature (input type matching the element, returning an int index in [0,n)).
2. Check the wrapped underlying error in the panic message for the exact marshaling failure.
3. Pass a plain named or literal function type rather than a dynamically built or exotic func value.

Example fix

// before: nil function
var fn func(int) int
beam.Partition(s, n, fn, col) // panics encoding partition function

// after
fn := func(e int) int { return e % n }
beam.Partition(s, n, fn, col)

if fn == nil {
    return errors.New("partition function must be non-nil")
}
// ensure fn is a func with signature func(T) int

defer func() { if r := recover(); r != nil { err = fmt.Errorf("partition fn encoding failed: %v", r) } }()

When it happens

Trigger: Passing a partition function whose signature/type cannot be encoded by reflectx.MakeFunc/EncodedFunc — e.g. a function value that is nil, or a fn argument whose reflection-based encoding fails.

Common situations: Passing nil as fn; passing a value that is not a function where a func is required; exotic function types not supported by Beam's reflection-based function encoding.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Could not unmarshal SourceConfig: %v
• n must be > 0
• unmarshalling partitionFn data
• empty graph
• trigger.AfterCount(%v) must be a positive integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/60c66a3ff9eaae3e. Report an issue: GitHub.