apache/beam · error · RuntimeError

The maximum number of retries has been reached

Error message

The maximum number of retries has been reached

What it means

Raised by wait_for_bq_job when the job has not reached DONE within max_retries polling iterations (each separated by sleep_duration_sec). It means Beam gave up waiting — the job is still running or stuck, not necessarily failed.

Solutions

1. Increase max_retries (or pass max_retries=0 to poll indefinitely) in the call site.
2. Increase sleep_duration_sec to reduce polling and allow more wall-clock time per retry.
3. Optimize the underlying query/export (partition filters, smaller columns, EXPORT with smaller output).
4. Check the job in BigQuery console to see if it is actually stuck or still making progress.

Example fix

// before
wrapper.wait_for_bq_job(job_ref, sleep_duration_sec=5, max_retries=10)  # 50s max

// after
wrapper.wait_for_bq_job(job_ref, sleep_duration_sec=60, max_retries=100)  # 100min max

# estimate runtime and size the poll budget accordingly
expected_seconds = estimate_job_runtime(table_bytes)
max_retries = max(1, int(expected_seconds / sleep_duration_sec) * 2)

def poll_budget_is_sufficient(max_retries, sleep_sec, expected_seconds):
    return max_retries == 0 or max_retries * sleep_sec >= expected_seconds

try:
    wrapper.wait_for_bq_job(job_ref, sleep_duration_sec=60, max_retries=100)
except RuntimeError as e:
    if 'maximum number of retries' in str(e):
        # job still running: re-attach polling or fail the pipeline with a timeout
        resume_wait_or_fail_with_timeout(job_ref)
    else:
        raise

When it happens

Trigger: wait_for_bq_job(job_reference, max_retries=N, sleep_duration_sec=S) where a long-running query/export exceeds N*S seconds; called by _execute_query and _export_files.

Common situations: Very large export via _export_files on huge tables; slow queries during slot contention; max_retries left at a small default while sleep_duration_sec is also small; jobs queued behind regional capacity.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• Unable to poll job status
• A BigQuery table or a query must be specified
• A function must be provided to convert the input type into…
• A schema is required in order to prepare rows for writing…
• A schema must be provided when writing to BigQuery using…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ea29fed6e8a78d78. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:707

      `RuntimeError`: If the job is FAILED or the number of retries has been
        reached.
    """
    retry = 0
    while True:
      retry += 1
      job = self.get_job(
          job_reference.projectId, job_reference.jobId, job_reference.location)
      _LOGGER.info('Job %s status: %s', job.id, job.status.state)
      if job.status.state == 'DONE' and job.status.errorResult:
        raise RuntimeError(
            'BigQuery job {} failed. Error Result: {}'.format(
                job_reference.jobId, job.status.errorResult))
      elif job.status.state == 'DONE':
        return True
      else:
        time.sleep(sleep_duration_sec)
        if max_retries != 0 and retry >= max_retries:
          raise RuntimeError('The maximum number of retries has been reached')

  @retry.with_exponential_backoff(
      num_retries=MAX_RETRIES,
      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
  def _get_query_results(
      self,
      project_id,
      job_id,
      page_token=None,
      max_results=10000,
      location=None):
    request = bigquery.BigqueryJobsGetQueryResultsRequest(
        jobId=job_id,
        pageToken=page_token,
        projectId=project_id,
        maxResults=max_results,
        location=location)
    response = self.client.jobs.GetQueryResults(request)

View on GitHub (pinned to 12126d8942)