{"record":{"id":"f8fce7b8f641ddb0","repo":"apache/beam","slug":"copy-job-failed","errorCode":null,"errorMessage":"Copy job {} failed.","messagePattern":"Copy job (.+?) failed\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java","lineNumber":356,"sourceCode":"              ? ref.getProjectId()\n              : loadJobProjectId.get();\n      BigQueryHelpers.PendingJob retryJob =\n          new BigQueryHelpers.PendingJob(\n              jobId -> {\n                JobReference jobRef =\n                    new JobReference()\n                        .setProjectId(projectId)\n                        .setJobId(jobId.getJobId())\n                        .setLocation(bqLocation);\n                LOG.info(\n                    \"Starting copy job for table {} using  {}, job id iteration {}\",\n                    ref,\n                    jobRef,\n                    jobId.getRetryIndex());\n                try {\n                  jobService.startCopyJob(jobRef, copyConfig);\n                } catch (IOException | InterruptedException e) {\n                  LOG.warn(\"Copy job {} failed.\", jobRef, e);\n                  throw new RuntimeException(e);\n                }\n                return null;\n              },\n              // Function to poll the result of a load job.\n              jobId -> {\n                JobReference jobRef =\n                    new JobReference()\n                        .setProjectId(projectId)\n                        .setJobId(jobId.getJobId())\n                        .setLocation(bqLocation);\n                try {\n                  return jobService.pollJob(jobRef, BatchLoads.LOAD_JOB_POLL_MAX_RETRIES);\n                } catch (InterruptedException e) {\n                  throw new RuntimeException(e);\n                }\n              },\n              // Function to lookup a job.","sourceCodeStart":338,"sourceCodeEnd":374,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java#L338-L374","documentation":"WriteRename.startCopy() failed to start the BigQuery copy job that moves temp-table contents into the final destination table: jobService.startCopyJob threw IOException or InterruptedException. The warning is logged with the job reference and the error, then a RuntimeException is thrown, aborting the write's finalize step (temp tables remain until cleanup).","triggerScenarios":"startCopyJob(jobRef, copyConfig) throws — permission denied on source or destination table, quota exceeded for copy jobs, invalid copy configuration (e.g. table in different region), or interruption during submission.","commonSituations":"Service account missing bigquery.jobs.create or dataEditor on the destination dataset; cross-region temp/destination tables; copy-job rate quota exceeded on large batch writes; transient Google API failures.","solutions":["Inspect the logged exception: grant roles/bigquery.jobUser + roles/bigquery.dataEditor if 403.","Ensure temp and destination tables are in the same region (createDisposition/location configuration).","If quota errors, reduce write parallelism or request higher copy-job quota.","Retry the pipeline; abandoned temp tables are cleaned up by the write's cleanup step or can be deleted manually."],"exampleFix":"// before: copy job 403 / cross-region failure\nBigQueryIO.writeTableRows().to(dest).withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)\n// after: pin job location so temp table and destination match\n.withLocation(\"US\") // same region as destination dataset","handlingStrategy":"retry","validationCode":"// pre-check: temp and destination datasets share a location\nDataset src = bq.getDataset(tmpDatasetId); Dataset dst = bq.getDataset(destDatasetId);\nassert src.getLocation().equals(dst.getLocation());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Grant bigquery.jobs.create and dataEditor on both temp and destination datasets.","Keep temp and destination tables in the same region/location.","Watch copy-job quotas on very large batch writes.","Clean up orphaned temp tables after failed runs."],"tags":["bigquery","copy-job","api","permissions"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}