apache/beam · error

SDK side reshuffle not yet supported

Error message

SDK side reshuffle not yet supported

What it means

handleReshuffle rejects reshuffles that the SDK has already expanded into backup transforms (h.config.SDKReshuffle true). Prism only implements its own runner-side reshuffle (a fusion break); it panics when asked to handle an SDK-side reshuffle expansion instead.

Solutions

1. Remove the SDKReshuffle setting (or set it false) so prism handles Reshuffle natively as a fusion break.
2. Rebuild the pipeline without SDK-side reshuffle expansion so the Reshuffle URN reaches the runner intact.
3. If SDK-side reshuffle is required, use a runner that supports it (e.g. Flink/Dataflow) or implement runner-side support in prism.

Example fix

// before
runnerOpts := []func(*prism.Option){ prism.SDKReshuffle(true) }
// after
runnerOpts := []func(*prism.Option){ prism.SDKReshuffle(false) }

// ensure prism options don't enable SDKReshuffle
if sdkReshuffleEnabled { /* route to a runner that supports SDK-side reshuffle */ }

defer func(){ if r := recover(); r != nil && strings.Contains(fmt.Sprint(r), "SDK side reshuffle not yet supported") { /* switch runner or config */ } }()

When it happens

Trigger: A pipeline is submitted with config.SDKReshuffle enabled while the graph contains a Reshuffle/Redistribute transform that was expanded on the SDK side into its backup group-by-key transforms.

Common situations: Enabling the SDKReshuffle prism config flag (usually intended to test SDK-side reshuffle behavior) in a normal pipeline run; combining an SDK build that expands reshuffle with prism's unsupported handling path.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Expected single input PCollection in reshuffle:
• Expected single output PCollection in reshuffle:
• expected CoGBK, got
• failed to expand Reshuffle transform for edge
• invalid pcollection

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/05b811967eaa49bc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/prism/internal/handlerunner.go:169

		// Return the new components which is the transforms consumer
		return prepareResult{
			// We sub this flatten with itself, to not drop it.
			SubbedComps: &pipepb.Components{
				Transforms:   tSubs,
				Pcollections: pcollSubs,
			},
			RemovedLeaves: nil,
			ForcedRoots:   forcedRoots,
		}
	}
	return prepareResult{}
}

func (h *runner) handleReshuffle(tid string, t *pipepb.PTransform, comps *pipepb.Components) prepareResult {
	// TODO: Implement the windowing strategy the "backup" transforms used for Reshuffle.

	if h.config.SDKReshuffle {
		panic("SDK side reshuffle not yet supported")
	}

	// A Reshuffle, in principle, is a no-op on the pipeline structure, WRT correctness.
	// It could however affect performance, so it exists to tell the runner that this
	// point in the pipeline needs a fusion break, to enable the pipeline to change it's
	// degree of parallelism.
	//
	// The change of parallelism goes both ways. It could allow for larger batch sizes
	// enable smaller batch sizes downstream if it is infact paralleizable.
	//
	// But for a single transform node per stage runner, we can elide it entirely,
	// since the input collection and output collection types match.

	// Get the input and output PCollections, there should only be 1 each.
	if len(t.GetInputs()) != 1 {
		panic("Expected single input PCollection in reshuffle: " + prototext.Format(t))
	}
	if len(t.GetOutputs()) != 1 {

View on GitHub (pinned to 12126d8942)