apache/beam · error · IOError

Failed to get the Dataflow job id.

Error message

Failed to get the Dataflow job id.

What it means

DataflowResult.wait_until_finish polls a background thread that needs a valid Dataflow job id. If the job was never successfully created/registered with the runner (self.has_job is False) while the pipeline was expected to be running, waiting is impossible and this IOError is raised.

Solutions

1. Inspect the job submission: confirm pipeline.run() succeeded and job_id() returns a value before waiting.
2. Fix underlying submission errors (project/region/credentials) — check earlier log output for the real API failure.
3. Retry job submission; Dataflow API flakiness at launch time is transient.
4. Wrap the wait in a has_job check: if result.has_job: result.wait_until_finish() else: re-run submission.

Example fix

// before
result = pipeline.run()
result.wait_until_finish()
// after
result = pipeline.run()
if result.has_job:
    result.wait_until_finish()
else:
    raise RuntimeError('Dataflow job submission failed; no job id')

if not result.has_job:
    raise RuntimeError('Dataflow job id missing; submission failed')

try:
    result.wait_until_finish()
except IOError as e:
    if 'Failed to get the Dataflow job id' in str(e):
        result = pipeline.run()  # resubmit; launch failed earlier
        result.wait_until_finish()

When it happens

Trigger: Calling result.wait_until_finish() on a Dataflow result object whose job id was never populated — e.g. job submission failed or was interrupted before the id was fetched, but the result object was still returned.

Common situations: Automated scripts that call pipeline.run().wait_until_finish() even when launch failed; flaky Dataflow API errors during job creation; credential/permission failures at submission time swallowed earlier.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• A schema is required to write non-schema'd data.
• A sink must inherit iobase.Sink, iobase.NativeSink, or be a…
• An explicit schema is required to write non-schema'd…
• Can not query metrics. Job id is unknown.
• Cannot call read after iterating.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6a52f791a14f0f75. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:805

    """
    if not self.has_job:
      # https://github.com/apache/beam/blob/8f71dc41b30a978095ca0e0699009e4f4445a618/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L867-L870
      return PipelineState.DONE

    self._update_job()

    return self._get_job_state()

  def is_in_terminal_state(self):
    if not self.has_job:
      return True

    return PipelineState.is_terminal(self._get_job_state())

  def wait_until_finish(self, duration=None):
    if not self.is_in_terminal_state():
      if not self.has_job:
        raise IOError('Failed to get the Dataflow job id.')
      gcp_options = self._options.view_as(GoogleCloudOptions)
      consoleUrl = (
          "Console URL: https://console.cloud.google.com/"
          f"dataflow/jobs/{gcp_options.region}/{self.job_id()}"
          f"?project={gcp_options.project}")
      thread = threading.Thread(
          target=DataflowRunner.poll_for_job_completion,
          args=(self._runner, self, duration))

      # Mark the thread as a daemon thread so a keyboard interrupt on the main
      # thread will terminate everything. This is also the reason we will not
      # use thread.join() to wait for the polling thread.
      thread.daemon = True
      thread.start()
      while thread.is_alive():
        time.sleep(5.0)

      # TODO: Merge the termination code in poll_for_job_completion and

View on GitHub (pinned to 12126d8942)