apache/beam · critical

unmarshalling partitionFn data

Error message

unmarshalling partitionFn data

What it means

makePartitionFn regenerates the partition DoFn on workers by JSON-unmarshaling previously serialized partitionData. If the encoded bytes cannot be unmarshaled, it panics with the context "unmarshalling partitionFn data". This is an internal deserialization failure of the transform's payload, usually caused by data corruption or a Beam version mismatch between job submission and worker.

Source

Thrown at sdks/go/pkg/beam/partition.go:180

	key := args[1]
	value := args[2]

	n := f.fnKV.Call2x1(key, value).(int)
	if n < 0 || n >= f.n {
		return []any{errors.Errorf("partitionFn(%v) = %v, want [0,%v)", value, n, f.n)}
	}

	emit := args[n+3]
	reflectx.MakeFunc3x0(emit).Call3x0(timestamp, key, value)

	var err error
	return []any{err}
}

func makePartitionFn(name string, t reflect.Type, enc []byte) reflectx.Func {
	var data partitionData
	if err := json.Unmarshal(enc, &data); err != nil {
		panic(errors.WithContext(err, "unmarshalling partitionFn data"))
	}
	if data.KV {
		return &partitionFnKV{
			name: name,
			t:    t,
			n:    data.N,
			fnKV: reflectx.ToFunc2x1(data.Fn.Fn),
		}
	}
	return &partitionFn{
		name: name,
		t:    t,
		n:    data.N,
		fn:   reflectx.ToFunc1x1(data.Fn.Fn),
	}
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Pin the same Apache Beam Go SDK version for submission and workers (matching container images, --workerHarnessContainerImage or environment config).
2. Clear stale staged artifacts/jar-less staging dirs and resubmit the job.
3. Reproduce locally with a direct runner; if it persists, inspect the panic's wrapped error for exact JSON offset/type failure and file a Beam issue with versions.

Example fix

// before: mismatched versions
// go.mod: github.com/apache/beam/sdks/v2 v2.45.0, workers built from v2.50.0

// after: align versions
go get github.com/apache/beam/sdks/v2@v2.50.0
go mod tidy
// rebuild worker image with the same SDK version

// pre-flight: verify SDK version consistency
// go.mod SDK version must match worker container image tag
if sdkVersion != workerImageVersion {
    log.Fatalf("SDK mismatch: go.mod=%s worker=%s", sdkVersion, workerImageVersion)
}

defer func() { if r := recover(); r != nil { err = fmt.Errorf("partitionFn deserialization failed: %v", r) } }()

When it happens

Trigger: Worker-side expansion of a Partition transform where the marshaled partitionData bytes are corrupted, truncated, or were produced by an incompatible Beam SDK version (schema changed between releases).

Common situations: Mixed SDK versions: pipeline submitted with one version of Beam Go SDK, workers/staging containers running another; corrupted staged artifacts; custom builds of the SDK diverging on partitionData fields.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Could not unmarshal SourceConfig: %v
• encoding partition function
• unable to json unmarshal --environment_config: %w
• could not unmarshal CoderRef from %v, failed to decode urn-l
• failed to marshal receiver %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/93324c691a50c4d0. Report an issue: GitHub.