apache/hadoop · error · IOException

Bucket doesn't match for source '%s' and destination '%s'!

Error message

Bucket doesn't match for source '%s' and destination '%s'!

What it means

composeObjects concatenates source objects into a destination object using the GCS compose API, which only works among objects in the same bucket. Before building the ComposeRequest the code validates that every source's bucket equals the destination's bucket and throws IOException("Bucket doesn't match for source '%s' and destination '%s'!") otherwise. This is a client-side precondition failure raised before any service call.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:373

      } else {
        if (ErrorTypeExtractor.getErrorType(e) == ErrorTypeExtractor.ErrorType.ALREADY_EXISTS) {
          throw (FileAlreadyExistsException)
              new FileAlreadyExistsException(
                  String.format("Object '%s' already exists.", resourceId)
              ).initCause(e);
        }
        throw new IOException(e);
      }
    }
  }

  GoogleCloudStorageItemInfo composeObjects(
      List<StorageResourceId> sources, StorageResourceId destination, CreateObjectOptions options)
      throws IOException {
    LOG.trace("composeObjects({}, {}, {})", sources, destination, options);
    for (StorageResourceId inputId : sources) {
      if (!destination.getBucketName().equals(inputId.getBucketName())) {
        throw new IOException(
            String.format(
                "Bucket doesn't match for source '%s' and destination '%s'!",
                inputId, destination));
      }
    }
    Storage.ComposeRequest request =
        Storage.ComposeRequest.newBuilder()
            .addSource(
                sources.stream().map(StorageResourceId::getObjectName).collect(Collectors.toList()))
            .setTarget(
                BlobInfo.newBuilder(destination.getBucketName(), destination.getObjectName())
                    .setContentType(options.getContentType())
                    .setContentEncoding(options.getContentEncoding())
                    .setMetadata(encodeMetadata(options.getMetadata()))
                    .build())
            .setTargetOptions(
                Storage.BlobTargetOption.generationMatch(
                    destination.hasGenerationId()

View on GitHub (pinned to 2add963021)

Solutions

Keep compose sources and destination in the same bucket
If inputs span buckets, first copy them into the destination bucket, then compose
Fix the job configuration so input/output paths share one bucket

Example fix

// before
composeObjects(
  Arrays.asList(new StorageResourceId("bucketA", "p1"), new StorageResourceId("bucketA", "p2")),
  new StorageResourceId("bucketB", "merged"));
  // -> Bucket doesn't match for source 'bucketA/p1' and destination 'bucketB/merged'!

// after
List<StorageResourceId> srcs = Arrays.asList(
  new StorageResourceId("bucketB", "p1"), new StorageResourceId("bucketB", "p2"));
composeObjects(srcs, new StorageResourceId("bucketB", "merged"));

Defensive patterns

Strategy: validation

Validate before calling

// Reject mixed-bucket compose inputs before calling the API
String dstBucket = destination.getBucketName();
for (StorageResourceId src : sources) {
  if (!dstBucket.equals(src.getBucketName())) {
    throw new IllegalArgumentException(
        "compose requires same-bucket sources: " + src);
  }
}
gcs.composeObjects(sources, destination, options);

Prevention

Keep concat/merge inputs and output on one bucket
Copy foreign-bucket inputs over before composing
Assert bucket equality in unit tests of compose call sites

When it happens

Trigger: Calling compose (directly or via the concat/append code paths that merge small files) where any source StorageResourceId has a different bucket name than the destination — e.g. concatenating gs://bucketA/part-* into gs://bucketB/merged.

Common situations: Small-files merge jobs configured with input and output on different buckets; hardcoding a destination bucket that differs from the input bucket; unit tests feeding mixed-bucket resource ids into compose.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ebb406320ed57c54. Report an issue: GitHub.