apache/beam · error

failed to expand Reshuffle transform for edge

Error message

failed to expand Reshuffle transform for edge: %v

What it means

expandReshuffle translates beam.Reshuffle into a GBK-with-union-coder + re-window pattern; the framework must construct window coders for the reshard to work with unbounded inputs. Failures constructing the union coder, re-windowing transforms, or their nodes are wrapped as 'failed to expand Reshuffle transform for edge: %v'.

Solutions

1. Check the wrapped inner error — usually makeUnionCoder or makeWindowCoder failure
2. Ensure the input uses a supported window kind (global/fixed/sliding/session), not a custom WindowFn
3. Replace beam.Reshuffle with an alternative resharding approach (e.g. GBK with a dummy key) if windowing is custom
4. Upgrade the Go SDK for newer windowing support

Example fix

null

// reshuffle only supports standard window kinds
if w.Kind != window.GlobalWindows && w.Kind != window.FixedWindows &&
   w.Kind != window.SlidingWindows && w.Kind != window.SessionWindows {
    return errors.New("reshuffle requires a standard window function")
}

func isSupportedWindow(w *window.Fn) bool {
    switch w.Kind {
    case window.GlobalWindows, window.FixedWindows, window.SlidingWindows, window.SessionWindows:
        return true
    }
    return false
}

if err := beam.Run(ctx, pr); err != nil {
    if strings.Contains(err.Error(), "failed to expand Reshuffle transform") {
        return fmt.Errorf("reshuffle incompatible with windowing: %w", err)
    }
    return err
}

When it happens

Trigger: Calling beam.Reshuffle on a PCollection whose windowing strategy cannot be marshalled into the union coder's window coder (makeWindowCoder failure), or where intermediate node creation fails.

Common situations: Reshuffle on PCollections with exotic/unsupported windowing (e.g. custom window functions); runner environments where window coders for unbounded streams can't be built; SDK version mismatches.

Related errors

• AfterProcessingTime trigger set without a delay or…
• attempted to add namespace to missing windowing strategy id
• could not unmarshal global window coder
• could not unmarshal window coder for pcollection
• could not unmarshal window coder for timer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ad04718e5a069728. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/graphx/translate.go:971

// In particular, the "backup plan" needs to:
//
//   - Encode the windowed element, preserving timestamps.
//   - Add random keys to the encoded windowed element []bytes
//   - GroupByKey (in the global window).
//   - Explode the resulting elements list.
//   - Decode the windowed element []bytes.
//
// While a simple reshard can be written in user terms, (timestamps and windows
// are accessible to user functions) there are some framework internal
// optimizations that can be done if the framework is aware of the reshard, though
// ideally this is handled on the runner side.
//
// User code is able to write reshards, but it's easier to access
// the window coders framework side, which is critical for the reshard
// to function with unbounded inputs.
func (m *marshaller) expandReshuffle(edge NamedEdge) (string, error) {
	handleErr := func(err error) (string, error) {
		return "", errors.Wrapf(err, "failed to expand Reshuffle transform for edge: %v", edge)
	}
	id := edgeID(edge.Edge)
	kvCoder, err := makeUnionCoder()
	if err != nil {
		return handleErr(err)
	}
	kvCoderID, err := m.coders.Add(kvCoder)
	if err != nil {
		return handleErr(err)
	}
	gbkCoderID, err := m.coders.Add(coder.NewCoGBK(kvCoder.Components))
	if err != nil {
		return handleErr(err)
	}

	var subtransforms []string

	in := edge.Edge.Input[0]

View on GitHub (pinned to 12126d8942)