apache/beam · error

Unable to find running job with name

Error message

Unable to find running job with name %s

What it means

GetRunningJobByName pages through Dataflow jobs in the given project/region filtering by name, looking for a job in a running state. If the pagination loop completes without matching a job, the function returns this error. It means no currently-running Dataflow job with that exact name exists in that project and region.

Solutions

  1. Verify the exact job name with `gcloud dataflow jobs list --filter="name=<name>" --region=<region>` and correct any typo.
  2. Confirm the project and region passed to GetRunningJobByName match where the job was actually launched.
  3. Check the job's state; if it is JOB_STATE_DONE/FAILED/CANCELLED it will not be found as a running job — re-launch or query by job ID instead.
  4. If jobs exist but pagination is interrupted, re-run; a transient API failure surfaces as a different error before reaching this line.

Example fix

// before
job, err := dataflowlib.GetRunningJobByName(ctx, service, project, "us-central1", "my_pipeline")
// after (verify actual name/region first, or look up by ID)
job, err := dataflowlib.GetRunningJobByName(ctx, service, project, "us-central1", "my-pipeline-0912")
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-check with gcloud or list API before attaching:
jobs, _ := dfService.Projects.Locations.Jobs.List(project, region).Filter("name=" + name).Do()
if len(jobs.Jobs) == 0 || jobs.Jobs[0].CurrentState != "JOB_STATE_RUNNING" {
    // skip attach, launch a new job instead
}

Try / catch

job, err := dataflowlib.GetRunningJobByName(ctx, service, project, region, name)
if err != nil && strings.Contains(err.Error(), "Unable to find running job") {
    // job not running: launch new or use cached job ID
    return launchNewJob(ctx, p)
}
if err != nil {
    return err
}

Prevention

When it happens

Trigger: Calling GetRunningJobByName (via Submit or Execute) with a job name that does not match any active job in the target project/region, or the job exists but is in a non-running state (done, failed, cancelled, updating) so it is filtered out.

Common situations: Attaching to a job after it already finished or was cancelled; a typo in the job name; querying the wrong GCP project or region; the job was launched in a different region than the one queried.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/12779f4586c6a37c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go:352

	jobsListCall := client.Projects.Locations.Jobs.List(project, region)
	jobsListCall.Filter("ACTIVE")
	jobsResponse, err := jobsListCall.Do()
	for {
		if err != nil {
			return nil, err
		}
		for _, job := range jobsResponse.Jobs {
			if job.Name == name {
				return job, nil
			}
		}
		if jobsResponse.NextPageToken == "" {
			break
		}
		jobsListCall.PageToken(jobsResponse.NextPageToken)
		jobsResponse, err = jobsListCall.Do()
	}
	return nil, errors.New(fmt.Sprintf("Unable to find running job with name %s", name))
}

// GetMetrics returns a collection of metrics describing the progress of a
// job by making a call to Cloud Monitoring service.
func GetMetrics(ctx context.Context, client *df.Service, project, region, jobID string) (*df.JobMetrics, error) {
	return client.Projects.Locations.Jobs.GetMetrics(project, region, jobID).Do()
}

// dataflowOptions provides Dataflow with non Go-specific pipeline options. These are the only
// pipeline options that are communicated to cross-language SDK harnesses, so any pipeline options
// needed for cross-language transforms in Dataflow must be declared here.
type dataflowOptions struct {
	Experiments                    []string `json:"experiments,omitempty"`
	PipelineURL                    string   `json:"pipelineUrl"`
	PipelineProtoHash              string   `json:"pipelineProtoHash,omitempty"`
	Region                         string   `json:"region"`
	TempLocation                   string   `json:"tempLocation"`
	DiskProvisionedIops            int64    `json:"diskProvisionedIops"`

View on GitHub (pinned to 12126d8942)