apache/beam · error

Schema update load job {} failed with {}

Error message

Schema update load job {} failed with {}

What it means

UpdateSchemaDestination's zero-byte load job (used to apply a schema update without writing rows) failed to start: jobService.startLoadJob threw IOException or InterruptedException. The code logs this warning and rethrows as RuntimeException, so the schema update step (and the write relying on it) fails.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/UpdateSchemaDestination.java:324

    BigQueryHelpers.PendingJob retryJob =
        new BigQueryHelpers.PendingJob(
            // Function to load the data.
            jobId -> {
              JobReference jobRef =
                  new JobReference()
                      .setProjectId(projectId)
                      .setJobId(jobId.getJobId())
                      .setLocation(bqLocation);
              LOG.info(
                  "Loading zero rows using job {}, job id {} iteration {}",
                  tableReference,
                  jobRef,
                  jobId.getRetryIndex());
              try {
                jobService.startLoadJob(
                    jobRef, loadConfig, new ByteArrayContent("text/plain", new byte[0]));
              } catch (IOException | InterruptedException e) {
                LOG.warn("Schema update load job {} failed with {}", jobRef, e.toString());
                throw new RuntimeException(e);
              }
              return null;
            },
            // Function to poll the result of a load job.
            jobId -> {
              JobReference jobRef =
                  new JobReference()
                      .setProjectId(projectId)
                      .setJobId(jobId.getJobId())
                      .setLocation(bqLocation);
              try {
                return jobService.pollJob(jobRef, BatchLoads.LOAD_JOB_POLL_MAX_RETRIES);
              } catch (InterruptedException e) {
                throw new RuntimeException(e);
              }
            },
            // Function to lookup a job.

View on GitHub (pinned to 12126d8942)

Solutions

1. Read e.toString(): if 403, grant roles/bigquery.jobUser and roles/bigquery.dataEditor to the service account.
2. If quota/rate errors, reduce parallel write destinations or request higher BigQuery load-job quota.
3. Verify the new schema is compatible with existing table data (no required-field narrowing, no unsupported type changes).
4. Retry on transient errors; check BigQuery job history for the failed jobRef for server-side details.

Example fix

// before: schema update fails with 403
// grant job submission rights
gcloud projects add-iam-policy-binding proj --member=serviceAccount:sa@proj.iam.gserviceaccount.com --role=roles/bigquery.jobUser
// after: startLoadJob succeeds

// pre-flight: ensure the service account can insert load jobs
bq.create(JobInfo.of(LoadJobConfiguration.of(tableId, "gs://nonexistent-probe"))); // expect quota/permission errors early

When it happens

Trigger: startLoadJob(jobRef, loadConfig, emptyContent) throws — invalid job configuration, quota exceeded, permission denied for jobs.insert, or interruption while submitting.

Common situations: Service account missing bigquery.jobs.create; too many load jobs in flight (quota); schema in loadConfig invalid for the table (e.g. incompatible field type change); transient Google API outage.

Related errors

• Failed to get table {} with {}
• Copy job {} failed.
• Unable to get project number
• Unable to confirm BigQuery %1$s presence for table "%2$s". I
• Encountered unsupported parameter(s) in read_gbq: {kwargs.ke

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/16c3fbf0bec5825e. Report an issue: GitHub.