{"record":{"id":"d1b6b04f998c6a3d","repo":"apache/beam","slug":"failed-to-split-datasource-at-index-v-last-index-v-at","errorCode":null,"errorMessage":"failed to split DataSource (at index: %v, last index: %v) at fraction %.4f with requested splits (%v indices from %v to %v)","messagePattern":"failed to split DataSource \\(at index: (.+?), last index: (.+?)\\) at fraction %\\.4f with requested splits \\((.+?) indices from (.+?) to (.+?)\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/go/pkg/beam/core/runtime/exec/datasource.go","lineNumber":748,"sourceCode":"\tfor _, s := range splits {\n\t\tif s >= safeStart && s <= endIdx {\n\t\t\tdiff := math.Abs(splitFloat - float64(s))\n\t\t\tif diff <= prevDiff {\n\t\t\t\tprevDiff = diff\n\t\t\t\tbestS = s\n\t\t\t} else {\n\t\t\t\tbreak // Stop early if the difference starts increasing.\n\t\t\t}\n\t\t}\n\t}\n\tif bestS != -1 {\n\t\treturn bestS, -1.0, nil\n\t}\n\t// Printing all splits is expensive. Instead, return the current start and\n\t// end indices, and fraction along with the range of the indices and how\n\t// many there are. This branch requires at least one split index, so we don't\n\t// need to bounds check the slice.\n\treturn -1, -1.0, fmt.Errorf(\"failed to split DataSource (at index: %v, last index: %v) at fraction %.4f with requested splits (%v indices from %v to %v)\",\n\t\tcurrIdx, endIdx, frac, len(splits), splits[0], splits[len(splits)-1])\n}\n\nfunc encodeElm(elm *FullValue, wc WindowEncoder, ec ElementEncoder) ([]byte, error) {\n\tvar b bytes.Buffer\n\tif err := EncodeWindowedValueHeader(wc, elm.Windows, elm.Timestamp, elm.Pane, &b); err != nil {\n\t\treturn nil, err\n\t}\n\tif err := ec.Encode(elm, &b); err != nil {\n\t\treturn nil, err\n\t}\n\treturn b.Bytes(), nil\n}\n\ntype concatReStream struct {\n\tfirst, next ReStream\n}\n","sourceCodeStart":730,"sourceCodeEnd":766,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/core/runtime/exec/datasource.go#L730-L766","documentation":"splitHelper could not find a valid split point matching the requested split indices/fraction for the DataSource. Instead of a wrong split, Beam refuses and reports the current index, end index, fraction, and the requested split range so the caller can diagnose why no split was possible.","triggerScenarios":"Calling Split with requested indices/fraction that cannot be honored given the source's current progress — e.g. requesting splits beyond the remaining elements, at the very start/end of the source, or with a fraction that maps outside valid ranges.","commonSituations":"Aggressive dynamic work rebalancing on nearly-finished or tiny bundles; SDF restrictions too small to split; runners requesting many splits on sources with few remaining elements.","solutions":["Retry with fewer or adjusted split indices, or a smaller fraction.","Check bundle size/remaining elements — splitting a nearly exhausted source is not possible; let it finish.","Review restriction providers so sub-element-splittable sources expose meaningful fractions.","Log currIdx/endIdx from the message to see whether the request was out of range."],"exampleFix":"// before: fixed number of splits regardless of source size\nres, err := source.Split(ctx, makeSplits(10), 0.5, bufSize)\n// after: scale requested splits to remaining work\nn := int64(math.Max(1, float64(bufSize)*0.5))\nres, err := source.Split(ctx, computeSplitIndices(n), 0.5, bufSize)\nif err != nil {\n    // fall back to no split and continue processing\n}","handlingStrategy":"validation","validationCode":"if frac < 0 || frac > 1 || len(splits) == 0 { return errors.New(\"split request out of valid range\") }","typeGuard":null,"tryCatchPattern":"res, err := source.Split(ctx, splits, frac, bufSize)\nif err != nil && strings.Contains(err.Error(), \"failed to split DataSource\") {\n    log.Printf(\"split not possible: %v; continuing without split\", err)\n    res = SplitResult{} // proceed unsplit\n}","preventionTips":["Scale requested split counts to remaining source size","Avoid splitting nearly exhausted bundles","Fall back to unsplit processing when the error occurs"],"tags":["beam-go","split","datasource","work-rebalancing"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}