{"record":{"id":"ea29fed6e8a78d78","repo":"apache/beam","slug":"the-maximum-number-of-retries-has-been-reached","errorCode":null,"errorMessage":"The maximum number of retries has been reached","messagePattern":"The maximum number of retries has been reached","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/bigquery_tools.py","lineNumber":707,"sourceCode":"      `RuntimeError`: If the job is FAILED or the number of retries has been\n        reached.\n    \"\"\"\n    retry = 0\n    while True:\n      retry += 1\n      job = self.get_job(\n          job_reference.projectId, job_reference.jobId, job_reference.location)\n      _LOGGER.info('Job %s status: %s', job.id, job.status.state)\n      if job.status.state == 'DONE' and job.status.errorResult:\n        raise RuntimeError(\n            'BigQuery job {} failed. Error Result: {}'.format(\n                job_reference.jobId, job.status.errorResult))\n      elif job.status.state == 'DONE':\n        return True\n      else:\n        time.sleep(sleep_duration_sec)\n        if max_retries != 0 and retry >= max_retries:\n          raise RuntimeError('The maximum number of retries has been reached')\n\n  @retry.with_exponential_backoff(\n      num_retries=MAX_RETRIES,\n      retry_filter=retry.retry_on_server_errors_and_timeout_filter)\n  def _get_query_results(\n      self,\n      project_id,\n      job_id,\n      page_token=None,\n      max_results=10000,\n      location=None):\n    request = bigquery.BigqueryJobsGetQueryResultsRequest(\n        jobId=job_id,\n        pageToken=page_token,\n        projectId=project_id,\n        maxResults=max_results,\n        location=location)\n    response = self.client.jobs.GetQueryResults(request)","sourceCodeStart":689,"sourceCodeEnd":725,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L689-L725","documentation":"Raised by wait_for_bq_job when the job has not reached DONE within max_retries polling iterations (each separated by sleep_duration_sec). It means Beam gave up waiting — the job is still running or stuck, not necessarily failed.","triggerScenarios":"wait_for_bq_job(job_reference, max_retries=N, sleep_duration_sec=S) where a long-running query/export exceeds N*S seconds; called by _execute_query and _export_files.","commonSituations":"Very large export via _export_files on huge tables; slow queries during slot contention; max_retries left at a small default while sleep_duration_sec is also small; jobs queued behind regional capacity.","solutions":["Increase max_retries (or pass max_retries=0 to poll indefinitely) in the call site.","Increase sleep_duration_sec to reduce polling and allow more wall-clock time per retry.","Optimize the underlying query/export (partition filters, smaller columns, EXPORT with smaller output).","Check the job in BigQuery console to see if it is actually stuck or still making progress."],"exampleFix":"// before\nwrapper.wait_for_bq_job(job_ref, sleep_duration_sec=5, max_retries=10)  # 50s max\n\n// after\nwrapper.wait_for_bq_job(job_ref, sleep_duration_sec=60, max_retries=100)  # 100min max","handlingStrategy":"retry","validationCode":"# estimate runtime and size the poll budget accordingly\nexpected_seconds = estimate_job_runtime(table_bytes)\nmax_retries = max(1, int(expected_seconds / sleep_duration_sec) * 2)","typeGuard":"def poll_budget_is_sufficient(max_retries, sleep_sec, expected_seconds):\n    return max_retries == 0 or max_retries * sleep_sec >= expected_seconds","tryCatchPattern":"try:\n    wrapper.wait_for_bq_job(job_ref, sleep_duration_sec=60, max_retries=100)\nexcept RuntimeError as e:\n    if 'maximum number of retries' in str(e):\n        # job still running: re-attach polling or fail the pipeline with a timeout\n        resume_wait_or_fail_with_timeout(job_ref)\n    else:\n        raise","preventionTips":["Size max_retries * sleep_duration_sec well above the worst-case job runtime","Use max_retries=0 for unbounded polling when your pipeline can tolerate it","Optimize long exports/queries (partition filters, column pruning) so they fit the budget","Alert on poll timeouts distinct from actual job failures"],"tags":["bigquery","timeout","polling","job"],"backgroundTag":"request-timeout","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}