What causes "Plan %v failed %v callbacks" in apache/beam?

Find which callback(s) failed by instrumenting/logging inside the registered callbacks

Plan failed callbacks

Error message

Plan %v failed %v callbacks

What it means

Plan.Finalize() runs each registered bundleFinalizer callback whose validity window (validUntil) has not expired and collects the indices of callbacks that failed. If any failed, it replaces the finalizer and returns an aggregate error naming the plan and the count of failed callbacks.

Solutions

Find which callback(s) failed by instrumenting/logging inside the registered callbacks
Fix the failing callback's resource access (connectivity, permissions) or make it tolerant of already-cleaned-up state
Check callback validUntil windows — expired callbacks are skipped, near-expiry ones may race; extend the window if appropriate
Retry the Finalize() call if the underlying operation is transient

Example fix

null

Defensive patterns

Strategy: try-catch

Validate before calling

// Make callbacks cheap and side-effect-tolerant before registration
bf.Callback(callbackID, validUntil, func() error { return cleanupIfNotAlreadyDone() })

Try / catch

if err := plan.Finalize(); err != nil {
    if strings.Contains(err.Error(), "failed") {
        // some callbacks failed; inspect callback logs, then optionally retry
        return plan.Finalize()
    }
    return err
}

Prevention

Make bundle callbacks idempotent (safe to re-run or already-done)
Keep callbacks fast and independent of external resources when possible
Set generous validUntil windows to avoid expiry races
Log per-callback errors inside the callback itself for diagnosability

When it happens

Trigger: One or more callbacks registered via bundleFinalizer return errors when invoked during Finalize() — e.g. an SDF checkpoint/cleanup callback (like a watermark or state cleanup) fails.

Common situations: Custom bundle finalizers whose callbacks touch external resources (files, DBs) that are unavailable at finalize time; callback validity windows raced with the cleanup; SDF checkpoint bookkeeping failing after long bundles.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3afd9fe81edfba35. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/plan.go:198

		}
	}

	newFinalizer := bundleFinalizer{
		callbacks:         []bundleFinalizationCallback{},
		lastValidCallback: time.Now(),
	}

	for _, idx := range failedIndices {
		newFinalizer.callbacks = append(newFinalizer.callbacks, p.bf.callbacks[idx])
		if newFinalizer.lastValidCallback.Before(p.bf.callbacks[idx].validUntil) {
			newFinalizer.lastValidCallback = p.bf.callbacks[idx].validUntil
		}
	}

	p.bf = &newFinalizer

	if len(failedIndices) > 0 {
		return errors.Errorf("Plan %v failed %v callbacks", p.ID(), len(failedIndices))
	}
	return nil
}

// GetExpirationTime returns the last expiration time of any of the callbacks registered by the bundleFinalizer.
// Once we have passed this time, it is safe to move this plan to inactive without missing any valid callbacks.
func (p *Plan) GetExpirationTime() time.Time {
	return p.bf.lastValidCallback
}

// Down takes the plan and associated units down. Does not panic.
func (p *Plan) Down(ctx context.Context) error {
	// Technically racy, but only one thread calls this method on the plan.
	if p.getStatus() == Down {
		return nil // ok: already down
	}
	p.setStatus(Down)

View on GitHub (pinned to 12126d8942)