{"record":{"id":"6a52f791a14f0f75","repo":"apache/beam","slug":"failed-to-get-the-dataflow-job-id","errorCode":null,"errorMessage":"Failed to get the Dataflow job id.","messagePattern":"Failed to get the Dataflow job id\\.","errorType":"exception","errorClass":"IOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dataflow/dataflow_runner.py","lineNumber":805,"sourceCode":"    \"\"\"\n    if not self.has_job:\n      # https://github.com/apache/beam/blob/8f71dc41b30a978095ca0e0699009e4f4445a618/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L867-L870\n      return PipelineState.DONE\n\n    self._update_job()\n\n    return self._get_job_state()\n\n  def is_in_terminal_state(self):\n    if not self.has_job:\n      return True\n\n    return PipelineState.is_terminal(self._get_job_state())\n\n  def wait_until_finish(self, duration=None):\n    if not self.is_in_terminal_state():\n      if not self.has_job:\n        raise IOError('Failed to get the Dataflow job id.')\n      gcp_options = self._options.view_as(GoogleCloudOptions)\n      consoleUrl = (\n          \"Console URL: https://console.cloud.google.com/\"\n          f\"dataflow/jobs/{gcp_options.region}/{self.job_id()}\"\n          f\"?project={gcp_options.project}\")\n      thread = threading.Thread(\n          target=DataflowRunner.poll_for_job_completion,\n          args=(self._runner, self, duration))\n\n      # Mark the thread as a daemon thread so a keyboard interrupt on the main\n      # thread will terminate everything. This is also the reason we will not\n      # use thread.join() to wait for the polling thread.\n      thread.daemon = True\n      thread.start()\n      while thread.is_alive():\n        time.sleep(5.0)\n\n      # TODO: Merge the termination code in poll_for_job_completion and","sourceCodeStart":787,"sourceCodeEnd":823,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L787-L823","documentation":"DataflowResult.wait_until_finish polls a background thread that needs a valid Dataflow job id. If the job was never successfully created/registered with the runner (self.has_job is False) while the pipeline was expected to be running, waiting is impossible and this IOError is raised.","triggerScenarios":"Calling result.wait_until_finish() on a Dataflow result object whose job id was never populated — e.g. job submission failed or was interrupted before the id was fetched, but the result object was still returned.","commonSituations":"Automated scripts that call pipeline.run().wait_until_finish() even when launch failed; flaky Dataflow API errors during job creation; credential/permission failures at submission time swallowed earlier.","solutions":["Inspect the job submission: confirm pipeline.run() succeeded and job_id() returns a value before waiting.","Fix underlying submission errors (project/region/credentials) — check earlier log output for the real API failure.","Retry job submission; Dataflow API flakiness at launch time is transient.","Wrap the wait in a has_job check: if result.has_job: result.wait_until_finish() else: re-run submission."],"exampleFix":"// before\nresult = pipeline.run()\nresult.wait_until_finish()\n// after\nresult = pipeline.run()\nif result.has_job:\n    result.wait_until_finish()\nelse:\n    raise RuntimeError('Dataflow job submission failed; no job id')","handlingStrategy":"try-catch","validationCode":"if not result.has_job:\n    raise RuntimeError('Dataflow job id missing; submission failed')","typeGuard":null,"tryCatchPattern":"try:\n    result.wait_until_finish()\nexcept IOError as e:\n    if 'Failed to get the Dataflow job id' in str(e):\n        result = pipeline.run()  # resubmit; launch failed earlier\n        result.wait_until_finish()","preventionTips":["Check result.has_job / job_id() before waiting","Log and handle exceptions from pipeline.run() itself","Retry submission on transient Dataflow launch failures"],"tags":["python","apache-beam","dataflow","job-submission","io"],"backgroundTag":"resource-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}