apache/beam · error · IOException

Unable to find BigQuery job: %s, aborting after %d retries.

Error message

Unable to find BigQuery job: %s, aborting after %d retries.

What it means

BigQueryIO throws this IOException in JobServiceImpl.getJob after MAX_RPC_RETRIES failed attempts to fetch a BigQuery job via Jobs.get. Each polling failure (IOException) is logged, recorded, and retried with backoff; once retries are exhausted the last exception is attached and thrown. It means the runner could not confirm the status of a BigQuery job (used for load/extract/copy jobs during pipeline execution).

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:586

              .execute();
        } catch (GoogleJsonResponseException e) {
          if (errorExtractor.itemNotFound(e)) {
            LOG.info(
                "No BigQuery job with job id {} found in location {}.",
                jobId,
                jobRef.getLocation());
            return null;
          }
          LOG.info(
              "Ignoring the error encountered while trying to query the BigQuery job {}", jobId, e);
          lastException = e;
        } catch (IOException e) {
          LOG.info(
              "Ignoring the error encountered while trying to query the BigQuery job {}", jobId, e);
          lastException = e;
        }
      } while (nextBackOff(sleeper, backoff));
      throw new IOException(
          String.format(
              "Unable to find BigQuery job: %s, aborting after %d retries.",
              jobRef, MAX_RPC_RETRIES),
          lastException);
    }

    @Override
    public void close() throws Exception {}
  }

  @VisibleForTesting
  public static class DatasetServiceImpl implements DatasetService {

    // Backoff: 200ms * 1.5 ^ n, n=[1,5]
    private static final FluentBackoff INSERT_BACKOFF_FACTORY =
        FluentBackoff.DEFAULT.withInitialBackoff(Duration.millis(200)).withMaxRetries(5);

    // A backoff for rate limit exceeded errors. Only retry up to approximately 2 minutes

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the attached lastException for the concrete API error (404 vs 403 vs 5xx)
Confirm the job id/project/location used in the jobRef matches where the job was actually created
Grant the service account BigQuery Job User/Viewer roles so Jobs.get is permitted
Check network connectivity and the BigQuery status page if the cause is transient (timeouts, 5xx)
If the job is genuinely gone (expired/deleted), restart the affected load/export step rather than polling

Example fix

// before: job created in US but polled without location, causing repeated lookup failures
options.setBigQueryLocation("US"); // after: set the correct job location so Jobs.get resolves the job

Defensive patterns

Strategy: retry

Validate before calling

// Verify the job is retrievable right after creation
Job poll = bigquery.jobs().get(projectId, jobId).setLocation(location).execute(); // location must match where job was created

Try / catch

try {
  Job job = jobService.getJob(jobRef);
} catch (IOException e) {
  Throwable cause = e.getCause();
  if (cause instanceof GoogleJsonResponseException) {
    int code = ((GoogleJsonResponseException) cause).getStatusCode();
    if (code == 404) throw new IllegalStateException("Job not found; check project/location", cause);
    if (code == 403) throw new SecurityException("Missing bigquery.jobs.get permission", cause);
  }
  throw e; // transient: retry the operation later
}

Prevention

Set the correct BigQuery location so job lookups resolve to the right region
Grant Job User/Viewer roles enabling bigquery.jobs.get on the project
Avoid deleting/expiring jobs that a running pipeline still polls
Add network monitoring for long batch runs that depend on job-status polling

When it happens

Trigger: Polling the status of an already-started BigQuery job (via JobService.getJob) when every Jobs.get RPC fails — job reference points to a job that cannot be read, permission denied on the project, or sustained API/network failure across all retries.

Common situations: Job was created in a different project/region than queried (location mismatch); credentials lack bigquery.jobs.get permission; the job was deleted or expired before polling; network partition or BigQuery outage lasting longer than the retry window; unit tests (testGetJobThrows) intentionally triggering this path.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c3817cc02580ab0f. Report an issue: GitHub.