apache/beam · error

failed to split at requested splits

Error message

failed to split at requested splits: {%v}, DataSource not initialized

What it means

DataSource.Split performs dynamic work rebalancing (liquid sharding). If the DataSource pointer itself is nil, there is no source to split, so Beam returns this error including the requested split indices. It is an internal precondition violation: splitting is requested against an uninitialized source.

Solutions

1. Ensure the DataSource is initialized (constructed and started) before handling split requests.
2. Check worker logs for earlier initialization errors that left the source nil.
3. Guard runner-side split handling: skip or defer splits until the source reports readiness.
4. Upgrade/verify the Beam Go SDK version for known split-during-startup races.

Example fix

// before
res, err := source.Split(ctx, splits, frac, bufSize)
// after
if source == nil {
    return SplitResult{}, nil // or retry after initialization
}
res, err := source.Split(ctx, splits, frac, bufSize)

if source == nil { return SplitResult{}, errors.New("source not initialized; cannot split") }

func splitable(n *exec.DataSource) bool { return n != nil }

res, err := source.Split(ctx, splits, frac, bufSize)
if err != nil && strings.Contains(err.Error(), "DataSource not initialized") {
    return handleUninitializedSource(ctx)
}

When it happens

Trigger: Calling Split (directly or via the runner's split handler) on a nil *DataSource, typically when bundle/source initialization failed or the source was never started before a split request arrived.

Common situations: Runner sends a split request before the DataSource is initialized or after teardown; custom runners driving exec.DataSource manually without calling Start; race between initialization failure and split RPC.

Related errors

• failed to split DataSource
• failed to split: splittable unit was nil
• cannot make a keyed iterable for an unkeyed side input
• cannot make a state provider for an unkeyed input
• capacity must be a positive integer, got

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4fa8e42f05cc0ea7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:554

}

// Split takes a sorted set of potential split indices and a fraction of the
// remainder to split at, selects and actuates a split on an appropriate split
// index, and returns the selected split index in a SplitResult if successful or
// an error when unsuccessful.
//
// If the following transform is splittable, and the split indices and fraction
// allow for splitting on the currently processing element, then a sub-element
// split is performed, and the appropriate information is returned in the
// SplitResult.
//
// The bufSize param specifies the estimated number of elements that will be
// sent to this DataSource, and is used to be able to perform accurate splits
// even if the DataSource has not yet received all its elements. A bufSize of
// 0 or less indicates that it's unknown, and so uses the current known size.
func (n *DataSource) Split(ctx context.Context, splits []int64, frac float64, bufSize int64) (SplitResult, error) {
	if n == nil {
		return SplitResult{}, fmt.Errorf("failed to split at requested splits: {%v}, DataSource not initialized", splits)
	}
	if frac > 1.0 {
		frac = 1.0
	} else if frac < 0.0 {
		frac = 0.0
	}

	n.mu.Lock()
	defer n.mu.Unlock()

	var currProg float64 // Current element progress.
	var su SplittableUnit
	if n.index < 0 { // Progress is at the end of the non-existant -1st element.
		currProg = 1.0
	} else if n.su == nil { // If this isn't sub-element splittable, estimate some progress.
		currProg = 0.5
	} else { // If this is sub-element splittable, get progress of the current element.

View on GitHub (pinned to 12126d8942)