{"record":{"id":"c3817cc02580ab0f","repo":"apache/beam","slug":"unable-to-find-bigquery-job-s-aborting-after-d-retries","errorCode":null,"errorMessage":"Unable to find BigQuery job: %s, aborting after %d retries.","messagePattern":"Unable to find BigQuery job: (.+?), aborting after (.+?) retries\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java","lineNumber":586,"sourceCode":"              .execute();\n        } catch (GoogleJsonResponseException e) {\n          if (errorExtractor.itemNotFound(e)) {\n            LOG.info(\n                \"No BigQuery job with job id {} found in location {}.\",\n                jobId,\n                jobRef.getLocation());\n            return null;\n          }\n          LOG.info(\n              \"Ignoring the error encountered while trying to query the BigQuery job {}\", jobId, e);\n          lastException = e;\n        } catch (IOException e) {\n          LOG.info(\n              \"Ignoring the error encountered while trying to query the BigQuery job {}\", jobId, e);\n          lastException = e;\n        }\n      } while (nextBackOff(sleeper, backoff));\n      throw new IOException(\n          String.format(\n              \"Unable to find BigQuery job: %s, aborting after %d retries.\",\n              jobRef, MAX_RPC_RETRIES),\n          lastException);\n    }\n\n    @Override\n    public void close() throws Exception {}\n  }\n\n  @VisibleForTesting\n  public static class DatasetServiceImpl implements DatasetService {\n\n    // Backoff: 200ms * 1.5 ^ n, n=[1,5]\n    private static final FluentBackoff INSERT_BACKOFF_FACTORY =\n        FluentBackoff.DEFAULT.withInitialBackoff(Duration.millis(200)).withMaxRetries(5);\n\n    // A backoff for rate limit exceeded errors. Only retry up to approximately 2 minutes","sourceCodeStart":568,"sourceCodeEnd":604,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L568-L604","documentation":"BigQueryIO throws this IOException in JobServiceImpl.getJob after MAX_RPC_RETRIES failed attempts to fetch a BigQuery job via Jobs.get. Each polling failure (IOException) is logged, recorded, and retried with backoff; once retries are exhausted the last exception is attached and thrown. It means the runner could not confirm the status of a BigQuery job (used for load/extract/copy jobs during pipeline execution).","triggerScenarios":"Polling the status of an already-started BigQuery job (via JobService.getJob) when every Jobs.get RPC fails — job reference points to a job that cannot be read, permission denied on the project, or sustained API/network failure across all retries.","commonSituations":"Job was created in a different project/region than queried (location mismatch); credentials lack bigquery.jobs.get permission; the job was deleted or expired before polling; network partition or BigQuery outage lasting longer than the retry window; unit tests (testGetJobThrows) intentionally triggering this path.","solutions":["Inspect the attached lastException for the concrete API error (404 vs 403 vs 5xx)","Confirm the job id/project/location used in the jobRef matches where the job was actually created","Grant the service account BigQuery Job User/Viewer roles so Jobs.get is permitted","Check network connectivity and the BigQuery status page if the cause is transient (timeouts, 5xx)","If the job is genuinely gone (expired/deleted), restart the affected load/export step rather than polling"],"exampleFix":"// before: job created in US but polled without location, causing repeated lookup failures\noptions.setBigQueryLocation(\"US\"); // after: set the correct job location so Jobs.get resolves the job","handlingStrategy":"retry","validationCode":"// Verify the job is retrievable right after creation\nJob poll = bigquery.jobs().get(projectId, jobId).setLocation(location).execute(); // location must match where job was created","typeGuard":null,"tryCatchPattern":"try {\n  Job job = jobService.getJob(jobRef);\n} catch (IOException e) {\n  Throwable cause = e.getCause();\n  if (cause instanceof GoogleJsonResponseException) {\n    int code = ((GoogleJsonResponseException) cause).getStatusCode();\n    if (code == 404) throw new IllegalStateException(\"Job not found; check project/location\", cause);\n    if (code == 403) throw new SecurityException(\"Missing bigquery.jobs.get permission\", cause);\n  }\n  throw e; // transient: retry the operation later\n}","preventionTips":["Set the correct BigQuery location so job lookups resolve to the right region","Grant Job User/Viewer roles enabling bigquery.jobs.get on the project","Avoid deleting/expiring jobs that a running pipeline still polls","Add network monitoring for long batch runs that depend on job-status polling"],"tags":["bigquery","gcp","retry-exhausted","job-polling"],"backgroundTag":"resource-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}