apache/beam · error

while executing Process for

Error message

while executing Process for %v

What it means

During Plan.Execute, each root unit's Process() step is run; if it (or user DoFn code beneath it) returns an error or panics, the plan is marked Broken and the error is wrapped with the plan identity. This indicates bundle processing failed mid-flight, so the plan can no longer be trusted and must be torn down or restarted.

Solutions

1. Look at the wrapped (inner) error for the actual failing unit/DoFn and fix that root cause
2. Make the DoFn idempotent/robust: validate input records and return descriptive errors or use a dead-letter output instead of failing
3. Retry the whole bundle: the plan is Broken, so build/re-execute a fresh Plan rather than reusing this one
4. Check whether the failing record is poison; guard with nil checks or schema validation in the DoFn

Example fix

// before: DoFn crashes on nil
func (f *myFn) ProcessElement(ctx context.Context, elm Customer, emit func(string)) error {
    return doWork(elm.PrimaryID) // panics if elm is zero-valued
}
// after
func (f *myFn) ProcessElement(ctx context.Context, elm Customer, emit func(string)) error {
    if elm.PrimaryID == "" {
        return nil // or emit to a dead-letter PCollection
    }
    return doWork(elm.PrimaryID)
}

// In the DoFn, validate before doing work
func (f *myFn) ProcessElement(elm Customer, emit func(string)) error {
    if elm.PrimaryID == "" { return nil }
    return nil
}

func isValidCustomer(c Customer) bool { return c.PrimaryID != "" }

if err := plan.Execute(ctx); err != nil {
    var wrapped interface{ Unwrap() error }
    log.Printf("plan broken: %v; cause: %v", err, errors.Unwrap(err))
    plan.Down() // must tear down; plan cannot be reused
    return fmt.Errorf("bundle failed: %w", err)
}

When it happens

Trigger: Any root unit returns an error from Process() during bundle execution — e.g. a DoFn's ProcessElement returns an error, an inner exec unit fails, or a panic is converted by callNoPanic.

Common situations: User DoFn code fails on a specific record (nil field, bad cast, network call inside the DoFn), a side input fails to load, or a sink write fails during a streaming bundle on Dataflow/Flink runners.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• plan failed
• Plan failed callbacks
• plan failed with multiple errors
• while executing FinishBundle for
• AfterProcessingTime trigger set without a delay or…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e6f5d20dab4b053d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/exec/plan.go:156

	}

	// Process bundle. If there are any kinds of failures, we bail and mark the plan broken.

	p.setStatus(Active)
	for _, root := range p.roots {
		if err := callNoPanic(ctx, func(ctx context.Context) error { return root.StartBundle(ctx, id, manager) }); err != nil {
			p.setStatus(Broken)
			return errors.Wrapf(err, "while executing StartBundle for %v", p)
		}
	}
	for _, root := range p.roots {
		if err := callNoPanic(ctx, func(ctx context.Context) error {
			cps, err := root.Process(ctx)
			p.checkpoints = cps
			return err
		}); err != nil {
			p.setStatus(Broken)
			return errors.Wrapf(err, "while executing Process for %v", p)
		}
	}
	for _, root := range p.roots {
		if err := callNoPanic(ctx, root.FinishBundle); err != nil {
			p.setStatus(Broken)
			return errors.Wrapf(err, "while executing FinishBundle for %v", p)
		}
	}
	p.setStatus(Up)
	return nil
}

// Finalize runs any callbacks registered by the bundleFinalizer. Should be run on bundle finalization.
func (p *Plan) Finalize() error {
	if s := p.getStatus(); s != Up {
		return errors.Errorf("invalid status for plan %v: %v", p.id, s)
	}
	failedIndices := []int{}

View on GitHub (pinned to 12126d8942)