apache/beam · error

Unable to poll job status

Error message

Unable to poll job status: {}, aborting after reached max .

What it means

BigQueryServicesImpl.JobServiceImpl.pollJob retries polling the BigQuery insert-all/extract/load job status up to JOB_POLL_MAX_RETRIES with exponential backoff. If every attempt fails (IOException) or the backoff is exhausted, it logs this warning and returns null, signaling the caller that job status could not be determined.

Solutions

1. Check the chained IOException logs ('Ignore the error and retry polling job status') for the underlying cause (permissions, quota, network).
2. Verify the service account has bigquery.jobs.get permission on the project.
3. Increase JOB_POLL_MAX_RETRIES or backoff if the job is long-running and transient failures are expected.
4. Retry the pipeline stage; the job may still complete server-side, so check the job in the BigQuery console via `bq show -j`.

// Pre-check API reachability and permissions before polling:
// bq show -j --format=prettyjson --project_id=<proj> <jobId>  (exit code 0 = OK)

try { Job job = jobService.pollJob(jobRef, MAX_RETRIES); if (job == null) { /* handle: query job status manually or fail stage */ } } catch (IOException e) { /* retry with backoff */ }

When it happens

Trigger: Calling pollJob on a jobRef while the BigQuery API repeatedly returns IOException for getJob/status requests until max retries and backoff are exhausted; the message ends with the job ID substituted at runtime.

Common situations: Transient BigQuery service outages or elevated 5xx rates; network connectivity issues from workers; project/job permission problems causing persistent API errors; quota exhaustion on the BigQuery API.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• The maximum number of retries has been reached
• Failed to get table with
• : Timeout while initializing partition ' '. Kafka client…
• Taking over 5 minutes to flush gcs op batches after error
• Timed out waiting for service after

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1a82cb233b1fb1fa. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:501

            LOG.info("Still waiting for BigQuery job {} to enter pending state", jobRef);
            continue;
          }
          if ("DONE".equals(status.getState())) {
            LOG.info("BigQuery job {} completed in state DONE", jobRef);
            return job;
          }
          // The job is not DONE, wait longer and retry.
          LOG.info(
              "Still waiting for BigQuery job {}, currently in status {}\n{}",
              jobRef.getJobId(),
              status,
              formatBqStatusCommand(jobRef.getProjectId(), jobRef.getJobId()));
        } catch (IOException e) {
          // ignore and retry
          LOG.info("Ignore the error and retry polling job status.", e);
        }
      } while (nextBackOff(sleeper, backoff));
      LOG.warn("Unable to poll job status: {}, aborting after reached max .", jobRef.getJobId());
      return null;
    }

    private static String formatBqStatusCommand(String projectId, String jobId) {
      return String.format("bq show -j --format=prettyjson --project_id=%s %s", projectId, jobId);
    }

    @Override
    public JobStatistics dryRunQuery(
        String projectId, JobConfigurationQuery queryConfig, @Nullable String location)
        throws InterruptedException, IOException {
      @SuppressWarnings("nullness") // setLocation is not annotated, but does accept nulls
      JobReference jobRef = new JobReference().setLocation(location).setProjectId(projectId);
      Job job =
          new Job()
              .setJobReference(jobRef)
              .setConfiguration(new JobConfiguration().setQuery(queryConfig).setDryRun(true));
      // Use a custom backoff to avoid blocking job submission on being able to do a dry run.

View on GitHub (pinned to 12126d8942)