apache/beam · error · DataflowJobAlreadyExistsError

The job named with id: has already been updated into job…

Error message

The job named %s with id: %s has already been updated into job id: %s and cannot be updated again.

What it means

DataflowJobAlreadyExistsError thrown during job submission when the service reports the job was already updated into a different job id. This happens only when --update is enabled: the server's client_request_id indicates the update request was already processed, so the same update cannot be replayed. Beam surfaces it so users know their update already succeeded under a new job id.

Solutions

Treat the new job id in the message as the active job — monitor that job instead of resubmitting.
Use a fresh job name (or remove --update) if you genuinely want a new job.
Check the Dataflow console/API for the existing job before resubmitting.

Example fix

// before
p.run()  # with --update, same job_name, retried after timeout
// after
# detect and reuse
try:
  result = p.run()
except DataflowJobAlreadyExistsError as e:
  logging.info('Job already updated: %s', e)  # resume with returned job id

Defensive patterns

Strategy: try-catch

Validate before calling

# before submitting with --update, check whether the named job was already updated
existing = [j for j in dataflow_client.list_jobs(project=proj, location=loc).jobs
            if j.name == job_name and j.current_state in ('JOB_STATE_RUNNING','JOB_STATE_DRAINING')]

Try / catch

try:
    result = pipeline.run()
except DataflowJobAlreadyExistsError as e:
    logging.info('Update already applied: %s', e)  # parse and reuse the new job id
    return

Prevention

Don't blindly retry --update submissions after timeouts; check job state first.
Use idempotent, unique job names per pipeline version.
Inspect the Dataflow console/API before resubmitting.

When it happens

Trigger: Submitting a pipeline with --update where the Dataflow service returns a response whose client_request_id differs from the newly generated one — i.e. the job with that name was already updated previously via a retried/submitted request.

Common situations: Re-running a pipeline that was already successfully updated; retrying a submission after a network timeout where the first attempt actually succeeded; CI jobs re-running with the same --job_name and --update.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/746a45325ca1f6af. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:914

    request.project_id = self.google_cloud_options.project
    request.location = self.google_cloud_options.region
    request.job = job.proto

    try:
      response = self._jobs_client.create_job(request=request)
    except exceptions.GoogleAPICallError as e:
      _LOGGER.error(
          'HTTP status %d trying to create job'
          ' at dataflow service endpoint %s',
          e.code,
          self.google_cloud_options.dataflow_endpoint)
      _LOGGER.fatal('details of server error: %s', e)
      raise

    if response.client_request_id and \
        response.client_request_id != job.proto.client_request_id:
      if self.google_cloud_options.update:
        raise DataflowJobAlreadyExistsError(
            "The job named %s with id: %s has already been updated into job "
            "id: %s and cannot be updated again." %
            (response.name, job.proto.replace_job_id, response.id))
      else:
        raise DataflowJobAlreadyExistsError(
            'There is already active job named %s with id: %s. If you want to '
            'submit a second job, try again by setting a different name using '
            '--job_name.' % (response.name, response.id))

    _LOGGER.info('Create job: %s', response)
    # The response is a Job proto with the id for the new job.
    _LOGGER.info('Created job with id: [%s]', response.id)
    _LOGGER.info('Submitted job: %s', response.id)
    _LOGGER.info(
        'To access the Dataflow monitoring console, please navigate to '
        'https://console.cloud.google.com/dataflow/jobs/%s/%s?project=%s',
        self.google_cloud_options.region,
        response.id,

View on GitHub (pinned to 12126d8942)