apache/beam · error

failed to submit job

Error message

failed to submit job

What it means

Submit starts a prepared job by calling the JobService Run RPC. If that RPC fails, the error is wrapped as "failed to submit job". Preparation succeeded, but the runner refused or failed to begin executing the job.

Solutions

1. Retry the whole submission (Prepare + Submit) — preparation ids are per-submission and may expire.
2. Inspect the wrapped gRPC status for the runner-side reason (quota, capacity, invalid state).
3. Verify artifact staging succeeded and the staging token is current.
4. Check runner service logs/health if failures persist.

for attempt := 0; attempt < 3; attempt++ {
    res, err := beam.Run(ctx, runner, p)
    if err != nil && strings.Contains(err.Error(), "failed to submit job") { time.Sleep(backoff); continue }
    break
}

When it happens

Trigger: client.Run(ctx, req) errors during Submit, called from Execute after a successful Prepare with a valid preparation id and staging token.

Common situations: Stale/expired preparation id or staging session token; runner overloaded or failing while starting workers; artifact staging not yet complete; transient gRPC errors between prepare and run.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• job failed to prepare
• chunk send failed
• error creating local job server
• failed to close stream for
• failed to connect to state service

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4bdf446f3b3e3dd9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/universal/runnerlib/job.go:93

		JobName:         opt.Name,
	}
	resp, err := client.Prepare(ctx, req)
	if err != nil {
		return "", "", "", errors.Wrap(err, "job failed to prepare")
	}
	return resp.GetPreparationId(), resp.GetArtifactStagingEndpoint().GetUrl(), resp.GetStagingSessionToken(), nil
}

// Submit submits a job to the given job service. It returns a jobID, if successful.
func Submit(ctx context.Context, client jobpb.JobServiceClient, id, token string) (string, error) {
	req := &jobpb.RunJobRequest{
		PreparationId:  id,
		RetrievalToken: token,
	}

	resp, err := client.Run(ctx, req)
	if err != nil {
		return "", errors.Wrap(err, "failed to submit job")
	}
	return resp.GetJobId(), nil
}

// WaitForCompletion monitors the given job until completion. It logs any messages
// and state changes received.
func WaitForCompletion(ctx context.Context, client jobpb.JobServiceClient, jobID string) error {
	stream, err := client.GetMessageStream(ctx, &jobpb.JobMessagesRequest{JobId: jobID})
	if err != nil {
		return errors.Wrap(err, "failed to get job stream")
	}

	mostRecentError := "<no error received>"
	var errReceived, jobFailed bool

	for {
		msg, err := stream.Recv()
		if err != nil {

View on GitHub (pinned to 12126d8942)