failed to split DataSource

Error message

failed to split DataSource (at index: %v, last index: %v) at fraction %.4f with requested splits (%v indices from %v to %v)

What it means

splitHelper could not find a valid split point matching the requested split indices/fraction for the DataSource. Instead of a wrong split, Beam refuses and reports the current index, end index, fraction, and the requested split range so the caller can diagnose why no split was possible.

Solutions

Retry with fewer or adjusted split indices, or a smaller fraction.
Check bundle size/remaining elements — splitting a nearly exhausted source is not possible; let it finish.
Review restriction providers so sub-element-splittable sources expose meaningful fractions.
Log currIdx/endIdx from the message to see whether the request was out of range.

Example fix

// before: fixed number of splits regardless of source size
res, err := source.Split(ctx, makeSplits(10), 0.5, bufSize)
// after: scale requested splits to remaining work
n := int64(math.Max(1, float64(bufSize)*0.5))
res, err := source.Split(ctx, computeSplitIndices(n), 0.5, bufSize)
if err != nil {
    // fall back to no split and continue processing
}

Defensive patterns

Strategy: validation

Validate before calling

if frac < 0 || frac > 1 || len(splits) == 0 { return errors.New("split request out of valid range") }

Try / catch

res, err := source.Split(ctx, splits, frac, bufSize)
if err != nil && strings.Contains(err.Error(), "failed to split DataSource") {
    log.Printf("split not possible: %v; continuing without split", err)
    res = SplitResult{} // proceed unsplit
}

Prevention

Scale requested split counts to remaining source size
Avoid splitting nearly exhausted bundles
Fall back to unsplit processing when the error occurs

When it happens

Trigger: Calling Split with requested indices/fraction that cannot be honored given the source's current progress — e.g. requesting splits beyond the remaining elements, at the very start/end of the source, or with a fraction that maps outside valid ranges.

Common situations: Aggressive dynamic work rebalancing on nearly-finished or tiny bundles; SDF restrictions too small to split; runners requesting many splits on sources with few remaining elements.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d1b6b04f998c6a3d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/datasource.go:748

	for _, s := range splits {
		if s >= safeStart && s <= endIdx {
			diff := math.Abs(splitFloat - float64(s))
			if diff <= prevDiff {
				prevDiff = diff
				bestS = s
			} else {
				break // Stop early if the difference starts increasing.
			}
		}
	}
	if bestS != -1 {
		return bestS, -1.0, nil
	}
	// Printing all splits is expensive. Instead, return the current start and
	// end indices, and fraction along with the range of the indices and how
	// many there are. This branch requires at least one split index, so we don't
	// need to bounds check the slice.
	return -1, -1.0, fmt.Errorf("failed to split DataSource (at index: %v, last index: %v) at fraction %.4f with requested splits (%v indices from %v to %v)",
		currIdx, endIdx, frac, len(splits), splits[0], splits[len(splits)-1])
}

func encodeElm(elm *FullValue, wc WindowEncoder, ec ElementEncoder) ([]byte, error) {
	var b bytes.Buffer
	if err := EncodeWindowedValueHeader(wc, elm.Windows, elm.Timestamp, elm.Pane, &b); err != nil {
		return nil, err
	}
	if err := ec.Encode(elm, &b); err != nil {
		return nil, err
	}
	return b.Bytes(), nil
}

type concatReStream struct {
	first, next ReStream
}

View on GitHub (pinned to 12126d8942)