{"record":{"id":"7a4503b3f5c27d0a","repo":"apache/beam","slug":"skipping-dest-existence-is-only-supported-within-a-bucket","errorCode":null,"errorMessage":"Skipping dest existence is only supported within a bucket.","messagePattern":"Skipping dest existence is only supported within a bucket\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":1270,"sourceCode":"      Iterable<String> srcFilenames,\n      Iterable<String> destFilenames,\n      boolean deleteSource,\n      boolean ignoreMissingSource,\n      boolean ignoreExistingDest)\n      throws IOException {\n    List<String> srcList = Lists.newArrayList(srcFilenames);\n    List<String> destList = Lists.newArrayList(destFilenames);\n    checkArgument(\n        srcList.size() == destList.size(),\n        \"Number of source files %s must equal number of destination files %s\",\n        srcList.size(),\n        destList.size());\n    LinkedList<RewriteOp> rewrites = Lists.newLinkedList();\n    for (int i = 0; i < srcList.size(); i++) {\n      final GcsPath sourcePath = GcsPath.fromUri(srcList.get(i));\n      final GcsPath destPath = GcsPath.fromUri(destList.get(i));\n      if (ignoreExistingDest && !sourcePath.getBucket().equals(destPath.getBucket())) {\n        throw new UnsupportedOperationException(\n            \"Skipping dest existence is only supported within a bucket.\");\n      }\n      rewrites.addLast(new RewriteOp(sourcePath, destPath, deleteSource, ignoreMissingSource));\n    }\n    return rewrites;\n  }\n\n  List<BatchInterface> makeRewriteBatches(LinkedList<RewriteOp> rewrites) throws IOException {\n    List<BatchInterface> batches = new ArrayList<>();\n    @Nullable BatchInterface opBatch = null;\n    boolean useSeparateRewriteDataBatch = this.rewriteDataOpBatchLimit != MAX_REQUESTS_PER_BATCH;\n    Iterator<RewriteOp> it = rewrites.iterator();\n    List<RewriteOp> deferredRewriteDataOps = new ArrayList<>();\n    while (it.hasNext()) {\n      RewriteOp rewrite = it.next();\n      if (!rewrite.getReadyToEnqueue()) {\n        it.remove();\n        continue;","sourceCodeStart":1252,"sourceCodeEnd":1288,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L1252-L1288","documentation":"Thrown by GcsUtilV1 when copying with ignoreExistingDest=true (skip if destination exists) while source and destination are in different GCS buckets. The skip-if-exists optimization relies on a same-bucket existence check via generation match, so cross-bucket copies with this flag are rejected as unsupported.","triggerScenarios":"Calling fileCopyRewrite (or filesystem copy) with the skip-destination-exists strategy where srcList.get(i).getBucket() differs from destList.get(i).getBucket() for any pair.","commonSituations":"Configuring a Beam FileSystems.copy with IGNORE behavior for cross-bucket renames/archives; moving staged files from a staging bucket to a final bucket while assuming skip-if-exists is allowed.","solutions":["Copy within the same bucket when using the ignore-existing-dest strategy.","Drop the ignore/skip strategy for cross-bucket copies and handle FileAlreadyExistsException explicitly.","Use OverwriteStrategy/OVERWRITE semantics if overwriting destinations is acceptable.","Pre-check destination existence yourself and filter the copy list before calling."],"exampleFix":"// before\noptions.copy(StandardCopyOption.IGNORE_EXISTING ...) cross-bucket\n// after: same-bucket only, or fall back to overwrite\nif (!src.getBucket().equals(dst.getBucket())) {\n  fs.copy(src, dst, StandardReplaceOptions.OVERWRITE /* or pre-check */);\n}","handlingStrategy":"validation","validationCode":"boolean sameBucket = srcs.stream()\n    .map(URI::toString)\n    .noneMatch(s -> !GcsPath.fromUri(s).getBucket()\n        .equals(GcsPath.fromUri(dstFor(s)).getBucket()));\nif (ignoreExistingDest && !sameBucket) {\n  // switch strategy or restrict to same-bucket copies\n}","typeGuard":null,"tryCatchPattern":"try {\n  copyWithSkip(...);\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage().contains(\"Skipping dest existence\")) {\n    copyWithOverwrite(...); // cross-bucket fallback\n  }\n}","preventionTips":["Only use skip-if-exists strategies for same-bucket copies","Design destination layouts so copy pairs stay within one bucket","Pre-check destination existence yourself for cross-bucket flows"],"tags":["gcs","unsupported-operation","cross-bucket","java","apache-beam"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}