apache/beam · critical

job %v failed: %w

Error message

job %v failed:
%w

What it means

The universal runner's WaitForCompletion streams JobMessages from the harness. When a message with severity >= JOB_MESSAGE_ERROR arrives and the job has also reached a FAILED state, it wraps the most recent error text as 'job %v failed:\n%w'. This is the standard surfaced error for a remote pipeline that died on the runner side.

Source

Thrown at sdks/go/pkg/beam/runners/universal/runnerlib/job.go:159

			var b strings.Builder
			if resp.GetTime() != "" {
				fmt.Fprintf(&b, "(time=%v)", resp.GetTime())
			}
			if resp.GetMessageId() != "" {
				fmt.Fprintf(&b, "(id=%v)", resp.GetMessageId())
			}
			b.WriteString(resp.GetMessageText())
			text := b.String()

			log.Output(ctx, messageSeverity(resp.GetImportance()), 1, text)

			if resp.GetImportance() >= jobpb.JobMessage_JOB_MESSAGE_ERROR {
				errReceived = true
				mostRecentError = resp.GetMessageText()

				if jobFailed {
					return errors.Errorf("job %v failed:\n%w", jobID, errors.New(mostRecentError))
				}
			}

		default:
			return errors.Errorf("unexpected job update: %v", msg.String())
		}
	}
}

func messageSeverity(importance jobpb.JobMessage_MessageImportance) log.Severity {
	switch importance {
	case jobpb.JobMessage_JOB_MESSAGE_ERROR:
		return log.SevError
	case jobpb.JobMessage_JOB_MESSAGE_WARNING:
		return log.SevWarn
	case jobpb.JobMessage_JOB_MESSAGE_BASIC:
		return log.SevInfo
	case jobpb.JobMessage_JOB_MESSAGE_DEBUG, jobpb.JobMessage_JOB_MESSAGE_DETAILED:

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped message after 'failed:' — it contains the actual harness error and fix that root cause
2. Check runner-side logs (worker logs) for stack traces around the failure time
3. Re-run with a local/direct runner first to reproduce user-code errors locally
4. Increase worker resources or container memory if the failure is OOM-related

Example fix

// before
if jobFailed {
	return errors.Errorf("job %v failed:\n%w", jobID, errors.New(mostRecentError))
}
// after
if jobFailed {
	return errors.Errorf("job %v failed:\n%s", jobID, mostRecentError) // inspect mostRecentError for root cause
}

if err := runner.WaitForCompletion(ctx, conn, jobID, deadline, printer); err != nil {
	if strings.HasPrefix(err.Error(), "job ") && strings.Contains(err.Error(), "failed:") {
		// parse text after 'failed:' for the harness root cause and log it
	}
	return err
}

When it happens

Trigger: Running a pipeline through the universal runner (Dataflow, Flink via portable runner) where the job transitions to FAILED and an error job message was received over the Lens/LegacyJobService stream.

Common situations: Worker crashes (OOM, missing dependencies in the worker container), user code panics in DoFns, runner-side resource exhaustion; the inner text carries the harness's actual error message.

Related errors

• could not detect user main
• unexpected job update: %v
• Type interface{} isn't a supported PCollection type
• Iterators with timestamp values (<ET,V> and <ET, K, V>) are
• Type interface{} isn't a supported PCollection type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fde60e71d7f1c06c. Report an issue: GitHub.