{"record":{"id":"0695d747f84f8223","repo":"apache/beam","slug":"error-completing-file-copies-with-retries-sample-from-s-to-s","errorCode":null,"errorMessage":"Error completing file copies with retries, sample: from %s to %s due to %s","messagePattern":"Error completing file copies with retries, sample: from (.+?) to (.+?) due to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":1223,"sourceCode":"      boolean ignoreMissingSource,\n      boolean ignoreExistingDest)\n      throws IOException {\n    LinkedList<RewriteOp> rewrites =\n        makeRewriteOps(\n            srcFilenames, destFilenames, deleteSource, ignoreMissingSource, ignoreExistingDest);\n    org.apache.beam.sdk.util.BackOff backoff = BACKOFF_FACTORY.backoff();\n    while (true) {\n      List<BatchInterface> batches = makeRewriteBatches(rewrites); // Removes completed rewrite ops.\n      if (batches.isEmpty()) {\n        break;\n      }\n      Preconditions.checkState(!rewrites.isEmpty());\n      RewriteOp sampleErrorOp =\n          rewrites.stream().filter(op -> op.getLastError() != null).findFirst().orElse(null);\n      if (sampleErrorOp != null) {\n        long backOffMillis = backoff.nextBackOffMillis();\n        if (backOffMillis == org.apache.beam.sdk.util.BackOff.STOP) {\n          throw new IOException(\n              String.format(\n                  \"Error completing file copies with retries, sample: from %s to %s due to %s\",\n                  sampleErrorOp.getFrom().toString(),\n                  sampleErrorOp.getTo().toString(),\n                  sampleErrorOp.getLastError()));\n        }\n        LOG.warn(\n            \"Retrying with backoff unsuccessful copy requests, sample request: from {} to {} due to {}\",\n            sampleErrorOp.getFrom(),\n            sampleErrorOp.getTo(),\n            sampleErrorOp.getLastError());\n        try {\n          Thread.sleep(backOffMillis);\n        } catch (InterruptedException e) {\n          Thread.currentThread().interrupt();\n          throw new IOException(\n              String.format(\n                  \"Interrupted backoff of file copies with retries, sample: from %s to %s due to %s\",","sourceCodeStart":1205,"sourceCodeEnd":1241,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L1205-L1241","documentation":"Thrown by GcsUtilV1 when a batch of GCS Rewrite (copy) operations keeps failing after exhausting the retry BackOff. The exception reports one representative failing rewrite ('sample') with its source path, destination path, and the last underlying error. It means GCS never finished the server-side copy within the allotted retry budget.","triggerScenarios":"Calling GcsUtilV1.fileCopyRewrite (via copy/move through the GCS filesystem layer) where some RewriteOp has a non-null lastError (e.g. 403, 404, rate-limit) and BackOff.nextBackOffMillis() returns STOP, meaning all retries are spent.","commonSituations":"Copying many large objects across buckets during pipeline staging; transient 5xx or rate limits that persist longer than the backoff window; source or destination objects deleted/permissions changed mid-copy; quota exhaustion on the GCS API.","solutions":["Inspect the sample from/to paths and underlying error in the message; fix the root cause (permissions, missing object) first.","Increase the retry BackOff budget (maxAttempts/maxDuration) configured for GCS operations in Beam options.","Re-run the pipeline; transient GCS errors often resolve on retry.","If copying across buckets, verify both buckets exist and the credentials have storage.objects.create on dest and get on source.","Consider using GcsUtilV2 (Storage-based) copy which handles overwrites differently."],"exampleFix":"// before: default small backoff exhausts on large batch copies\nGcsOptions options = ...; // gcsPipelineOptions.getGcpCredential etc.\n// after: enlarge retry budget\noptions.setGcsRewriteMaxRetrySeconds(600);\n// or catch and retry at a higher level\ntry {\n  gcsUtil.fileCopyRewrite(srcs, dsts);\n} catch (IOException e) {\n  // log sample path from message, then re-run with fresh backoff\n}","handlingStrategy":"retry","validationCode":"// check objects reachable and bucket permissions before batch copy\nfor (URI src : srcs) {\n  if (!gcsUtil.bucketAccessible(GcsPath.fromUri(src).getBucket())) {\n    throw new IllegalArgumentException(\"Inaccessible bucket: \" + src);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  gcsUtil.fileCopyRewrite(srcs, dsts);\n} catch (IOException e) {\n  if (e.getMessage().startsWith(\"Error completing file copies\")) {\n    // re-enqueue failed pair after fixing root cause / with longer backoff\n  }\n}","preventionTips":["Configure generous GCS retry backoff budgets for large batch copies","Pre-validate source/dest bucket access and object existence before the batch","Watch for sustained 429/5xx rates and reduce concurrency"],"tags":["gcs","retry-exhausted","io","java","apache-beam"],"backgroundTag":"retry-exhausted","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}