apache/beam · error · IOException

Error completing file copies with retries, sample: from %s t

Error message

Error completing file copies with retries, sample: from %s to %s due to %s

What it means

Thrown by GcsUtilV1 when a batch of GCS Rewrite (copy) operations keeps failing after exhausting the retry BackOff. The exception reports one representative failing rewrite ('sample') with its source path, destination path, and the last underlying error. It means GCS never finished the server-side copy within the allotted retry budget.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:1223

      boolean ignoreMissingSource,
      boolean ignoreExistingDest)
      throws IOException {
    LinkedList<RewriteOp> rewrites =
        makeRewriteOps(
            srcFilenames, destFilenames, deleteSource, ignoreMissingSource, ignoreExistingDest);
    org.apache.beam.sdk.util.BackOff backoff = BACKOFF_FACTORY.backoff();
    while (true) {
      List<BatchInterface> batches = makeRewriteBatches(rewrites); // Removes completed rewrite ops.
      if (batches.isEmpty()) {
        break;
      }
      Preconditions.checkState(!rewrites.isEmpty());
      RewriteOp sampleErrorOp =
          rewrites.stream().filter(op -> op.getLastError() != null).findFirst().orElse(null);
      if (sampleErrorOp != null) {
        long backOffMillis = backoff.nextBackOffMillis();
        if (backOffMillis == org.apache.beam.sdk.util.BackOff.STOP) {
          throw new IOException(
              String.format(
                  "Error completing file copies with retries, sample: from %s to %s due to %s",
                  sampleErrorOp.getFrom().toString(),
                  sampleErrorOp.getTo().toString(),
                  sampleErrorOp.getLastError()));
        }
        LOG.warn(
            "Retrying with backoff unsuccessful copy requests, sample request: from {} to {} due to {}",
            sampleErrorOp.getFrom(),
            sampleErrorOp.getTo(),
            sampleErrorOp.getLastError());
        try {
          Thread.sleep(backOffMillis);
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
          throw new IOException(
              String.format(
                  "Interrupted backoff of file copies with retries, sample: from %s to %s due to %s",

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the sample from/to paths and underlying error in the message; fix the root cause (permissions, missing object) first.
Increase the retry BackOff budget (maxAttempts/maxDuration) configured for GCS operations in Beam options.
Re-run the pipeline; transient GCS errors often resolve on retry.
If copying across buckets, verify both buckets exist and the credentials have storage.objects.create on dest and get on source.
Consider using GcsUtilV2 (Storage-based) copy which handles overwrites differently.

Example fix

// before: default small backoff exhausts on large batch copies
GcsOptions options = ...; // gcsPipelineOptions.getGcpCredential etc.
// after: enlarge retry budget
options.setGcsRewriteMaxRetrySeconds(600);
// or catch and retry at a higher level
try {
  gcsUtil.fileCopyRewrite(srcs, dsts);
} catch (IOException e) {
  // log sample path from message, then re-run with fresh backoff
}

Defensive patterns

Strategy: retry

Validate before calling

// check objects reachable and bucket permissions before batch copy
for (URI src : srcs) {
  if (!gcsUtil.bucketAccessible(GcsPath.fromUri(src).getBucket())) {
    throw new IllegalArgumentException("Inaccessible bucket: " + src);
  }
}

Try / catch

try {
  gcsUtil.fileCopyRewrite(srcs, dsts);
} catch (IOException e) {
  if (e.getMessage().startsWith("Error completing file copies")) {
    // re-enqueue failed pair after fixing root cause / with longer backoff
  }
}

Prevention

Configure generous GCS retry backoff budgets for large batch copies
Pre-validate source/dest bucket access and object existence before the batch
Watch for sustained 429/5xx rates and reduce concurrency

When it happens

Trigger: Calling GcsUtilV1.fileCopyRewrite (via copy/move through the GCS filesystem layer) where some RewriteOp has a non-null lastError (e.g. 403, 404, rate-limit) and BackOff.nextBackOffMillis() returns STOP, meaning all retries are spent.

Common situations: Copying many large objects across buckets during pipeline staging; transient 5xx or rate limits that persist longer than the backoff window; source or destination objects deleted/permissions changed mid-copy; quota exhaustion on the GCS API.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0695d747f84f8223. Report an issue: GitHub.