apache/beam · error

Copy job {} failed.

Error message

Copy job {} failed.

What it means

WriteRename.startCopy() failed to start the BigQuery copy job that moves temp-table contents into the final destination table: jobService.startCopyJob threw IOException or InterruptedException. The warning is logged with the job reference and the error, then a RuntimeException is thrown, aborting the write's finalize step (temp tables remain until cleanup).

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java:356

              ? ref.getProjectId()
              : loadJobProjectId.get();
      BigQueryHelpers.PendingJob retryJob =
          new BigQueryHelpers.PendingJob(
              jobId -> {
                JobReference jobRef =
                    new JobReference()
                        .setProjectId(projectId)
                        .setJobId(jobId.getJobId())
                        .setLocation(bqLocation);
                LOG.info(
                    "Starting copy job for table {} using  {}, job id iteration {}",
                    ref,
                    jobRef,
                    jobId.getRetryIndex());
                try {
                  jobService.startCopyJob(jobRef, copyConfig);
                } catch (IOException | InterruptedException e) {
                  LOG.warn("Copy job {} failed.", jobRef, e);
                  throw new RuntimeException(e);
                }
                return null;
              },
              // Function to poll the result of a load job.
              jobId -> {
                JobReference jobRef =
                    new JobReference()
                        .setProjectId(projectId)
                        .setJobId(jobId.getJobId())
                        .setLocation(bqLocation);
                try {
                  return jobService.pollJob(jobRef, BatchLoads.LOAD_JOB_POLL_MAX_RETRIES);
                } catch (InterruptedException e) {
                  throw new RuntimeException(e);
                }
              },
              // Function to lookup a job.

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the logged exception: grant roles/bigquery.jobUser + roles/bigquery.dataEditor if 403.
2. Ensure temp and destination tables are in the same region (createDisposition/location configuration).
3. If quota errors, reduce write parallelism or request higher copy-job quota.
4. Retry the pipeline; abandoned temp tables are cleaned up by the write's cleanup step or can be deleted manually.

Example fix

// before: copy job 403 / cross-region failure
BigQueryIO.writeTableRows().to(dest).withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
// after: pin job location so temp table and destination match
.withLocation("US") // same region as destination dataset

// pre-check: temp and destination datasets share a location
Dataset src = bq.getDataset(tmpDatasetId); Dataset dst = bq.getDataset(destDatasetId);
assert src.getLocation().equals(dst.getLocation());

When it happens

Trigger: startCopyJob(jobRef, copyConfig) throws — permission denied on source or destination table, quota exceeded for copy jobs, invalid copy configuration (e.g. table in different region), or interruption during submission.

Common situations: Service account missing bigquery.jobs.create or dataEditor on the destination dataset; cross-region temp/destination tables; copy-job rate quota exceeded on large batch writes; transient Google API failures.

Related errors

• Failed to get table {} with {}
• Schema update load job {} failed with {}
• Unable to get project number
• Unable to confirm BigQuery %1$s presence for table "%2$s". I
• Encountered unsupported parameter(s) in read_gbq: {kwargs.ke

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f8fce7b8f641ddb0. Report an issue: GitHub.