apache/beam · warning

Load job failed with

Error message

Load job {} failed with {}

What it means

A BigQuery load job (started via startLoadJob to bulk-load files into a BigQuery table) failed with an IOException or InterruptedException. The code logs the failure and rethrows it wrapped in a RuntimeException so the BigQueryIO retry logic can attempt the load again with a new retry index. This is part of WriteTables' batch-load path.

Solutions

1. Inspect the wrapped cause (e.getCause()) for the BigQuery API error details and fix the underlying issue (schema, format, permissions)
2. Check the service account has BigQuery Job User and Data Editor roles on the target dataset
3. Retry the pipeline; BigQueryIO automatically retries load jobs with an incremented retry index
4. Verify source URIs/files are readable and match the configured load format

Example fix

// before
throw new RuntimeException(e);
// after
// keep the wrap (needed for DoFn), but log the cause clearly:
LOG.error("BigQuery load job {} failed", jobRef, e.getCause());
throw new RuntimeException("BigQuery load job failed: " + jobRef.getJobId(), e);

// pre-check before starting the load job
throwIf(!gcsUris.stream().allMatch(FileSystems::match), "source URIs unreachable");
throwIf(!svcAccountHasRole("roles/bigquery.jobUser"), "missing BigQuery Job User role");

try {
  jobService.startLoadJob(jobRef, loadConfig);
} catch (IOException e) {
  if (isRetryable(e)) throw e; // let BigQueryIO retry logic handle it
  throw new RuntimeException("non-retryable load failure: " + jobRef.getJobId(), e);
} catch (InterruptedException e) {
  Thread.currentThread().interrupt();
  throw new RuntimeException("load job interrupted", e);
}

When it happens

Trigger: The call jobService.startLoadJob(jobRef, loadConfig) throws IOException (BigQuery API error: quota exceeded, invalid schema/source format, table not found, permission denied) or InterruptedException (thread interrupted during API call).

Common situations: Load jobs failing due to malformed CSV/Avro/JSON source files, BigQuery rate limits, service account lacking bigquery.jobs.create, transient 500s from the BigQuery API, or pipeline cancellation interrupting the worker.

Related errors

• Unable to create dataset
• Append to stream failed with invalid offset of
• Append to stream failed with Status Code . The stream may…
• AvroRowWriter is not readable
• BigQuery source must be split before being read

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/373f5e9c0d05c39b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java:534

    PendingJob retryJob =
        new PendingJob(
            // Function to load the data.
            jobId -> {
              JobReference jobRef =
                  new JobReference()
                      .setProjectId(projectId)
                      .setJobId(jobId.getJobId())
                      .setLocation(bqLocation);
              LOG.info(
                  "Loading {} files into {} using job {}, job id iteration {}",
                  gcsUris.size(),
                  ref,
                  jobRef,
                  jobId.getRetryIndex());
              try {
                jobService.startLoadJob(jobRef, loadConfig);
              } catch (IOException | InterruptedException e) {
                LOG.warn("Load job {} failed with {}", jobRef, e.toString());
                throw new RuntimeException(e);
              }
              return null;
            },
            // Function to poll the result of a load job.
            jobId -> {
              JobReference jobRef =
                  new JobReference()
                      .setProjectId(projectId)
                      .setJobId(jobId.getJobId())
                      .setLocation(bqLocation);
              try {
                return jobService.pollJob(jobRef, BatchLoads.LOAD_JOB_POLL_MAX_RETRIES);
              } catch (InterruptedException e) {
                throw new RuntimeException(e);
              }
            },
            // Function to lookup a job.

View on GitHub (pinned to 12126d8942)