What causes "job %v failed: %v" in apache/beam?

Read the %v detail (mostRecentError) — it usually names the failing step and root exception.

job %v failed: %v

Error message

job %v failed:
%v

What it means

While streaming job messages, WaitForCompletion records the most recent error and whether the job entered a failed state. When the stream ends (io.EOF) and the job had failed, it returns this error containing the job id and the most recent error message reported by the runner. This is the standard way a remote job's failure surfaces to the pipeline author.

Source

Thrown at sdks/go/pkg/beam/runners/universal/runnerlib/job.go:115

// WaitForCompletion monitors the given job until completion. It logs any messages
// and state changes received.
func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error {
	stream, err := client.GetMessageStream(ctx, &jobpb.JobMessagesRequest{JobId: jobID})
	if err != nil {
		return errors.Wrap(err, "failed to get job stream")
	}

	mostRecentError := "<no error received>"
	var errReceived, jobFailed bool

	for {
		msg, err := stream.Recv()
		if err != nil {
			if err == io.EOF {
				if jobFailed {
					// Connection finished, so time to exit, produce what we have.
					return errors.Errorf("job %v failed:\n%v", jobID, mostRecentError)
				}
				return nil
			}
			return err
		}

		switch {
		case msg.GetStateResponse() != nil:
			resp := msg.GetStateResponse()

			log.Infof(ctx, "Job[%v] state: %v", jobID, resp.GetState().String())

			switch resp.State {
			case jobpb.JobState_DONE, jobpb.JobState_CANCELLED:
				return nil
			case jobpb.JobState_FAILED:
				jobFailed = true
				if errReceived {

View on GitHub (pinned to 12126d8942)

Solutions

Read the %v detail (mostRecentError) — it usually names the failing step and root exception.
Inspect full worker logs via the runner's UI or job logs endpoint using the job id.
Fix the user code / resource issue that caused the step to fail (null pointers, OOM, hot keys).
Resubmit the pipeline once the cause is addressed.

Defensive patterns

Strategy: try-catch

Try / catch

if err := beam.Run(ctx, runner, p); err != nil {
    var jobErr interface{ Error() string }
    if errors.As(err, &jobErr) && strings.Contains(jobErr.Error(), "failed:") {
        // parse the job id and embedded runner error, then fetch full logs
    }
}

Prevention

Add defensive error handling in DoFns so transient data issues don't fail the job.
Watch worker memory/CPU (OOM and hot keys are common job killers).
Dry-run pipelines on the direct runner with sample data before remote submission.
Correlate the job id in the message with the runner UI's log viewer for the full stack trace.

When it happens

Trigger: The message stream closes with io.EOF after the runner reported the job FAILED; WaitForCompletion returns errors.Errorf("job %v failed:\n%v", jobID, mostRecentError).

Common situations: User code in a DoFn panicked or errored during execution; worker crashed or OOMed; runner-side infrastructure failure; pipeline graph rejected at runtime.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aed33cc9ee3c0f24. Report an issue: GitHub.