apache/beam · error

error decoding buckets: %w

Error message

error decoding buckets: %w

What it means

After starting the $bucketAuto aggregation, getBuckets drains the cursor with cursor.All(ctx, &buckets). If iterating or BSON-decoding the bucket documents into the internal bucket struct fails, this error is wrapped as 'error decoding buckets'. The aggregation ran but its results could not be materialized.

Source

Thrown at sdks/go/pkg/beam/io/mongodbio/id_range_split.go:125

		bson.D{{
			Key: "$bucketAuto",
			Value: bson.M{
				"groupBy": "$_id",
				"buckets": count,
			},
		}},
	}

	opts := options.Aggregate().SetAllowDiskUse(true)

	cursor, err := collection.Aggregate(ctx, pipeline, opts)
	if err != nil {
		return nil, fmt.Errorf("error executing bucketAuto aggregation: %w", err)
	}

	var buckets []bucket
	if err := cursor.All(ctx, &buckets); err != nil {
		return nil, fmt.Errorf("error decoding buckets: %w", err)
	}

	return buckets, nil
}

func idRangesFromBuckets(buckets []bucket, outerRange idRange) []idRange {
	if len(buckets) == 0 {
		return nil
	}

	ranges := make([]idRange, len(buckets))

	for i := 0; i < len(buckets); i++ {
		subRange := idRange{}

		if i == 0 {
			subRange.MinInclusive = outerRange.MinInclusive
			subRange.Min = outerRange.Min

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped cause; decode errors indicate an unexpected response shape — check driver/server version compatibility
2. Increase cursor batch lifetime / keep-alive to avoid cursor timeouts on slow servers
3. Retry the pipeline; a transient network drop during cursor iteration often succeeds on a second run
4. Confirm the connected driver version matches the server (e.g. use a recent mongo-driver for newer servers)
5. Consider the splitVector strategy if bucketAuto repeatedly fails to materialize

Example fix

// before
require.NoError(t, err) // production code failed on cursor timeout
// after — server-side: raise cursor idle timeout
// db.adminCommand({setParameter: 1, cursorTimeoutMillis: 600000})
splits, err := bucketAutoSplits(ctx, collection, r, numSplits, bundleSize)

// ensure cursor survives iteration
var alive struct{ CursorTimeoutMillis int }
db.RunCommand(ctx, bson.D{{Key:"getParameter", Value:1}, {Key:"cursorTimeoutMillis", Value:1}}).Decode(&alive)

buckets, err := getBuckets(ctx, coll, pipeline)
if err != nil && strings.Contains(err.Error(), "cursor") {
	buckets, err = retryWithBackoff(ctx, getBuckets, coll, pipeline)
}

When it happens

Trigger: cursor.All fails mid-iteration: server connection drops while streaming results, a bucket document has an unexpected shape the driver cannot decode into the bucket struct, or the context is canceled during iteration.

Common situations: Cursor timeout / cursor not found on long-running aggregations against busy servers; driver/server version mismatch producing fields the decoder rejects; network flakiness during large result streaming during split planning.

Related errors

• error executing bucketAuto aggregation: %w
• error decoding document ID: %w
• error decoding document: %w
• No data
• fraction must be between 0 and 1

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/061f692f57495b3b. Report an issue: GitHub.