What causes "failed to get job" in apache/beam?

Retry the pipeline run or add retry/backoff around job monitoring; single poll failures are often transient

failed to get job

Error message

failed to get job

What it means

WaitForCompletion polls the Dataflow API (projects.locations.jobs.get) in a loop until the job reaches a terminal state. If any individual polling request fails, the error is wrapped with 'failed to get job' and returned, terminating monitoring. This means the SDK could not read the job's current status.

Solutions

Retry the pipeline run or add retry/backoff around job monitoring; single poll failures are often transient
Confirm the job still exists and the project/region match where it was launched
Check the service account retains dataflow.jobs.get permission for the job's lifetime
Inspect the wrapped cause's HTTP status: 404 → job deleted/wrong region, 403 → permissions, 5xx/timeout → transient

Defensive patterns

Strategy: retry

Try / catch

if err := run(); err != nil && strings.Contains(err.Error(), "failed to get job") {
    // inspect wrapped status: 404 job gone, 403 perms, 5xx transient
    log.Printf("job monitoring failed: %v", err)
}

Prevention

Use retry/backoff for job status polling instead of failing on the first error
Keep credentials valid for the full job duration (long streaming jobs)
Confirm the job is not deleted externally while the pipeline waits
Pre-verify project and region before launching

When it happens

Trigger: Running a pipeline to completion (Execute calls WaitForCompletion) when a jobs.Get poll fails: transient API 5xx, network drop, deleted job, wrong region, or revoked permissions mid-run.

Common situations: Long-running streaming jobs hitting a transient Google API outage during polling, the job being deleted via console/gcloud while the pipeline waits, credentials expiring during very long jobs, or the region being wrong after a job move.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/df0ae7f0840edf3e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go:270

		if err != nil {
			return nil, err
		}
		job.ReplaceJobId = runningJob.Id
	}
	upd, err := client.Projects.Locations.Jobs.Create(project, region, job).Do()
	if err == nil {
		log.Infof(ctx, "Submitted job: %v", upd.Id)
	}
	return upd, err
}

// WaitForCompletion monitors the given job until completion. It logs any messages
// and state changes received.
func WaitForCompletion(ctx context.Context, client *df.Service, project, region, jobID string) error {
	for {
		j, err := client.Projects.Locations.Jobs.Get(project, region, jobID).Do()
		if err != nil {
			return errors.Wrap(err, "failed to get job")
		}

		terminal, msg, err := currentStateMessage(j.CurrentState, jobID)
		if err != nil {
			return err
		}
		log.Infof(ctx, "%s", msg)
		if terminal {
			return nil
		}

		time.Sleep(30 * time.Second)
	}
}

// currentStateMessage indicates if the state is terminal, and provides a message to log, or an error.
// Errors are always terminal.
func currentStateMessage(currentState, jobID string) (bool, string, error) {

View on GitHub (pinned to 12126d8942)