apache/beam · error

failed to split: splittable unit was nil

Error message

failed to split: splittable unit was nil

What it means

When a DataSource processes sub-element-splittable elements, the current splittable unit is passed over the n.su channel. If a splittable unit is received but is nil, Beam cannot determine split progress and fails. This indicates the pipeline handed a nil splittable unit where a live one was expected.

Solutions

1. Retry the split; this can be a transient race between element completion and the split request.
2. Check the Splittable DoFn's restriction/size functions never produce nil or zero-size restrictions.
3. Verify the custom splitter's CreateRestricter/RestrictionTracker implementations are correct for the element type.
4. Upgrade the Beam Go SDK; several split-progress races have been fixed over time.

Example fix

// before
su = <-n.su // assumes non-nil
// after (caller-side resilience)
res, err := source.Split(ctx, splits, frac, bufSize)
if err != nil && strings.Contains(err.Error(), "splittable unit was nil") {
    time.Sleep(50 * time.Millisecond)
    res, err = source.Split(ctx, splits, frac, bufSize) // retry after element completes
}

select {
case su := <-n.su:
    if su == nil { return errors.New("no active splittable element; retry later") }
default:
}

func hasActiveSplittableUnit(ch chan *SplittableUnit) bool { select { case su := <-ch: return su != nil; default: return false } }

res, err := source.Split(ctx, splits, frac, bufSize)
if err != nil && strings.Contains(err.Error(), "splittable unit was nil") {
    time.Sleep(100 * time.Millisecond)
    res, err = source.Split(ctx, splits, frac, bufSize)
}

When it happens

Trigger: Calling Split on a source whose current element is sub-element-splittable, but the splittable unit channel yields nil (element finished processing concurrently, or the source produced a nil su).

Common situations: Race between element completion and split request; Splittable DoFn implementations that emit nil restrictions; splitting during the boundary instant when an element just finished.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• failed to split at requested splits
• failed to split DataSource
• param rangeEndEstimator cannot be nil. Implementing…
• cannot make a keyed iterable for an unkeyed side input
• cannot make a state provider for an unkeyed input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/502fc897c50152bf. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:577

		frac = 0.0
	}

	n.mu.Lock()
	defer n.mu.Unlock()

	var currProg float64 // Current element progress.
	var su SplittableUnit
	if n.index < 0 { // Progress is at the end of the non-existant -1st element.
		currProg = 1.0
	} else if n.su == nil { // If this isn't sub-element splittable, estimate some progress.
		currProg = 0.5
	} else { // If this is sub-element splittable, get progress of the current element.

		select {
		case su = <-n.su:
			// If an element is processing, we'll get a splittable unit.
			if su == nil {
				return SplitResult{}, fmt.Errorf("failed to split: splittable unit was nil")
			}
			defer func() {
				n.su <- su
			}()
			currProg = su.GetProgress()
		case <-time.After(500 * time.Millisecond):
			// Otherwise, the current element hasn't started processing yet
			// or has already finished. By adding a short timeout, we avoid
			// the first possibility, and can assume progress is at max.
			currProg = 1.0
		}
	}
	// Size to split within is the minimum of bufSize or splitIdx so we avoid
	// including elements we already know won't be processed.
	if bufSize <= 0 || n.splitIdx < bufSize {
		bufSize = n.splitIdx
	}
	s, fr, err := splitHelper(n.index, bufSize, currProg, splits, frac, su != nil)

View on GitHub (pinned to 12126d8942)