apache/beam · error · DataflowJobAlreadyExistsError

There is already active job named

Error message

There is already active job named %s with id: %s. If you want to submit a second job, try again by setting a different name using --job_name.

What it means

DataflowJobAlreadyExistsError thrown at submission when an active job with the same name already exists and --update was not requested. Dataflow requires job names to be unique among active jobs, so a plain submit colliding with a running job is rejected. Beam raises this with guidance to change the name or use --update.

Solutions

Pick a unique job name, e.g. append a timestamp: --job_name=my-job-$(date +%Y%m%d-%H%M%S).
Pass --update to update the existing job instead of creating a new one.
Cancel or drain the existing job before resubmitting with the same name.

Example fix

// before
python -m my_pipeline --job_name=my-job   # my-job already active
// after
python -m my_pipeline --job_name=my-job-$(date +%Y%m%d-%H%M%S)
# or add --update to replace the existing job

Defensive patterns

Strategy: try-catch

Validate before calling

jobs = dataflow_client.list_jobs(project=proj, location=loc).jobs
if any(j.name == job_name and j.current_state in ('JOB_STATE_RUNNING','JOB_STATE_DRAINING') for j in jobs):
    raise RuntimeError(f'Job {job_name} is already active')

Try / catch

try:
    result = pipeline.run()
except DataflowJobAlreadyExistsError as e:
    logging.error('Job name collision: %s', e)
    sys.exit(2)  # caller retries with a fresh name

Prevention

Generate job names with a timestamp or run id suffix.
Cancel/drain old jobs before reusing names in scheduled pipelines.
Avoid hardcoded job names shared across teams.

When it happens

Trigger: Submitting a pipeline whose --job_name matches an existing RUNNING/DRAINING Dataflow job while the update option is off.

Common situations: Hardcoded job_name in a script run twice; scheduled jobs where the previous run is still draining; two developers launching the same named pipeline.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a59dd430d7c7f89c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:919

      response = self._jobs_client.create_job(request=request)
    except exceptions.GoogleAPICallError as e:
      _LOGGER.error(
          'HTTP status %d trying to create job'
          ' at dataflow service endpoint %s',
          e.code,
          self.google_cloud_options.dataflow_endpoint)
      _LOGGER.fatal('details of server error: %s', e)
      raise

    if response.client_request_id and \
        response.client_request_id != job.proto.client_request_id:
      if self.google_cloud_options.update:
        raise DataflowJobAlreadyExistsError(
            "The job named %s with id: %s has already been updated into job "
            "id: %s and cannot be updated again." %
            (response.name, job.proto.replace_job_id, response.id))
      else:
        raise DataflowJobAlreadyExistsError(
            'There is already active job named %s with id: %s. If you want to '
            'submit a second job, try again by setting a different name using '
            '--job_name.' % (response.name, response.id))

    _LOGGER.info('Create job: %s', response)
    # The response is a Job proto with the id for the new job.
    _LOGGER.info('Created job with id: [%s]', response.id)
    _LOGGER.info('Submitted job: %s', response.id)
    _LOGGER.info(
        'To access the Dataflow monitoring console, please navigate to '
        'https://console.cloud.google.com/dataflow/jobs/%s/%s?project=%s',
        self.google_cloud_options.region,
        response.id,
        self.google_cloud_options.project)

    return response

  @retry.with_exponential_backoff()  # Using retry defaults from utils/retry.py

View on GitHub (pinned to 12126d8942)