apache/beam · error · UnsupportedOperationException

Skipping dest existence is only supported within a bucket.

Error message

Skipping dest existence is only supported within a bucket.

What it means

Thrown by GcsUtilV1 when copying with ignoreExistingDest=true (skip if destination exists) while source and destination are in different GCS buckets. The skip-if-exists optimization relies on a same-bucket existence check via generation match, so cross-bucket copies with this flag are rejected as unsupported.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:1270

      Iterable<String> srcFilenames,
      Iterable<String> destFilenames,
      boolean deleteSource,
      boolean ignoreMissingSource,
      boolean ignoreExistingDest)
      throws IOException {
    List<String> srcList = Lists.newArrayList(srcFilenames);
    List<String> destList = Lists.newArrayList(destFilenames);
    checkArgument(
        srcList.size() == destList.size(),
        "Number of source files %s must equal number of destination files %s",
        srcList.size(),
        destList.size());
    LinkedList<RewriteOp> rewrites = Lists.newLinkedList();
    for (int i = 0; i < srcList.size(); i++) {
      final GcsPath sourcePath = GcsPath.fromUri(srcList.get(i));
      final GcsPath destPath = GcsPath.fromUri(destList.get(i));
      if (ignoreExistingDest && !sourcePath.getBucket().equals(destPath.getBucket())) {
        throw new UnsupportedOperationException(
            "Skipping dest existence is only supported within a bucket.");
      }
      rewrites.addLast(new RewriteOp(sourcePath, destPath, deleteSource, ignoreMissingSource));
    }
    return rewrites;
  }

  List<BatchInterface> makeRewriteBatches(LinkedList<RewriteOp> rewrites) throws IOException {
    List<BatchInterface> batches = new ArrayList<>();
    @Nullable BatchInterface opBatch = null;
    boolean useSeparateRewriteDataBatch = this.rewriteDataOpBatchLimit != MAX_REQUESTS_PER_BATCH;
    Iterator<RewriteOp> it = rewrites.iterator();
    List<RewriteOp> deferredRewriteDataOps = new ArrayList<>();
    while (it.hasNext()) {
      RewriteOp rewrite = it.next();
      if (!rewrite.getReadyToEnqueue()) {
        it.remove();
        continue;

View on GitHub (pinned to 12126d8942)

Solutions

  1. Copy within the same bucket when using the ignore-existing-dest strategy.
  2. Drop the ignore/skip strategy for cross-bucket copies and handle FileAlreadyExistsException explicitly.
  3. Use OverwriteStrategy/OVERWRITE semantics if overwriting destinations is acceptable.
  4. Pre-check destination existence yourself and filter the copy list before calling.

Example fix

// before
options.copy(StandardCopyOption.IGNORE_EXISTING ...) cross-bucket
// after: same-bucket only, or fall back to overwrite
if (!src.getBucket().equals(dst.getBucket())) {
  fs.copy(src, dst, StandardReplaceOptions.OVERWRITE /* or pre-check */);
}
Defensive patterns

Strategy: validation

Validate before calling

boolean sameBucket = srcs.stream()
    .map(URI::toString)
    .noneMatch(s -> !GcsPath.fromUri(s).getBucket()
        .equals(GcsPath.fromUri(dstFor(s)).getBucket()));
if (ignoreExistingDest && !sameBucket) {
  // switch strategy or restrict to same-bucket copies
}

Try / catch

try {
  copyWithSkip(...);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("Skipping dest existence")) {
    copyWithOverwrite(...); // cross-bucket fallback
  }
}

Prevention

When it happens

Trigger: Calling fileCopyRewrite (or filesystem copy) with the skip-destination-exists strategy where srcList.get(i).getBucket() differs from destList.get(i).getBucket() for any pair.

Common situations: Configuring a Beam FileSystems.copy with IGNORE behavior for cross-bucket renames/archives; moving staged files from a staging bucket to a final bucket while assuming skip-if-exists is allowed.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7a4503b3f5c27d0a. Report an issue: GitHub.