generating bundle for stage %v at watermark %v panicked %v

Error message

generating bundle for stage %v at watermark %v panicked
%v

What it means

startEventTimeBundle wraps bundle creation for event-time-triggered aggregation stages in a recover, and re-panics with the stage ID and watermark attached. This is not an independent failure but a context wrapper: any panic inside buildEventTimeBundle (e.g. the zero-length key panics or hold-tracker panics) surfaces as this message. The root cause is the wrapped inner panic text appended after \n%v.

Source

Thrown at sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:1877

	ss.mu.Lock()
	defer ss.mu.Unlock()
	return ss.output
}

// TODO: Move to better place for configuration
var (
	OneKeyPerBundle  bool // OneKeyPerBundle sets if a bundle is restricted to a single key.
	OneElementPerKey bool // OneElementPerKey sets if a key in a bundle is restricted to one element.
)

// startBundle initializes a bundle with elements if possible.
// A bundle only starts if there are elements at all, and if it's
// an aggregation stage, if the windowing stratgy allows it.
// Returns a non-zero adjustment to the pending elements count if the stage is accumulating.
func (ss *stageState) startEventTimeBundle(watermark mtime.Time, genBundID func() string) (string, bool, bool, int) {
	defer func() {
		if e := recover(); e != nil {
			panic(fmt.Sprintf("generating bundle for stage %v at watermark %v panicked\n%v", ss.ID, watermark, e))
		}
	}()
	ss.mu.Lock()
	defer ss.mu.Unlock()
	toProcess, minTs, newKeys, holdsInBundle, panesInBundle, stillSchedulable, accumulatingPendingAdjustment := ss.kind.buildEventTimeBundle(ss, watermark)

	if len(toProcess) == 0 {
		// If we have nothing, there's nothing to progress.
		return "", false, stillSchedulable, accumulatingPendingAdjustment
	}

	bundID := ss.makeInProgressBundle(genBundID, toProcess, minTs, newKeys, holdsInBundle, panesInBundle)
	slog.Debug("started an event time bundle", "stageID", ss.ID, "bundleID", bundID, "bundleSize", len(toProcess), "upstreamWatermark", watermark)

	return bundID, true, stillSchedulable, accumulatingPendingAdjustment
}

// buildEventTimeBundle for ordinary stages processes all pending elements.

View on GitHub (pinned to 12126d8942)

Solutions

Read the wrapped inner panic after \n in the message — fix that root cause first (often 'zero length key' or hold tracker errors)
Inspect pending elements for the named stage for empty keys before the aggregation stage
Verify trigger/watermark configuration for the stage; misaligned triggering can expose latent data issues
If reproducible on stock pipelines, capture the full panic and report it to the Beam project

Defensive patterns

Strategy: try-catch

Try / catch

// Parse the wrapped inner panic to find the root cause
parts := strings.SplitN(msg, "\n", 2)
if len(parts) == 2 {
    rootCause := parts[1]
    log.Printf("bundle panic root cause: %s", rootCause)
}

Prevention

Always read the wrapped panic text after the newline for the real cause
Fix the underlying zero-length-key / hold errors, not this wrapper
Keep trigger and watermark configurations conservative in tests

When it happens

Trigger: Any panic inside ss.kind.buildEventTimeBundle — most commonly zero-length keys in pending elements, or hold-count corruption — while starting an event-time bundle at the given watermark.

Common situations: Streaming pipelines with event-time triggers/aggregation where upstream stages emitted malformed elements (empty keys) or where watermark hold bookkeeping got out of sync.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/55e7ca0ddfbb5a0e. Report an issue: GitHub.