What causes "Bucket not found: %s" in apache/hadoop?

Correct the source bucket name in the path/configuration

apache/hadoop · error · FileNotFoundException

Bucket not found: %s

Error message

Bucket not found: %s

What it means

validateCopyArguments checks buckets before a cross-bucket copy: it resolves the source bucket's info (through a bucketInfoCache) and, if that bucket does not exist, throws FileNotFoundException("Bucket not found: " + srcBucketName). The check exists because cross-bucket copies require bucket metadata (location, storage class) which cannot be fetched for a nonexistent bucket.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:827

    if (sourceToDestinationObjectsMap.isEmpty()) {
      return;
    }

    Map<StorageResourceId, GoogleCloudStorageItemInfo> bucketInfoCache = new HashMap<>();

    for (Map.Entry<StorageResourceId, StorageResourceId> entry :
        sourceToDestinationObjectsMap.entrySet()) {
      StorageResourceId source = entry.getKey();
      StorageResourceId destination = entry.getValue();
      String srcBucketName = source.getBucketName();
      String dstBucketName = destination.getBucketName();
      // Avoid copy across locations or storage classes.
      if (!srcBucketName.equals(dstBucketName)) {
        StorageResourceId srcBucketResourceId = new StorageResourceId(srcBucketName);
        GoogleCloudStorageItemInfo srcBucketInfo =
            getGoogleCloudStorageItemInfo(gcsImpl, bucketInfoCache, srcBucketResourceId);
        if (!srcBucketInfo.exists()) {
          throw new FileNotFoundException("Bucket not found: " + srcBucketName);
        }

        StorageResourceId dstBucketResourceId = new StorageResourceId(dstBucketName);
        GoogleCloudStorageItemInfo dstBucketInfo =
            getGoogleCloudStorageItemInfo(gcsImpl, bucketInfoCache, dstBucketResourceId);
        if (!dstBucketInfo.exists()) {
          throw new FileNotFoundException("Bucket not found: " + dstBucketName);
        }

        // TODO: Restrict this only when copy-with-rewrite is enabled
        if (!srcBucketInfo.getLocation().equals(dstBucketInfo.getLocation())) {
          throw new UnsupportedOperationException(
              "This operation is not supported across two different storage locations.");
        }

        if (!srcBucketInfo.getStorageClass().equals(dstBucketInfo.getStorageClass())) {
          throw new UnsupportedOperationException(
              "This operation is not supported across two different storage classes.");

View on GitHub (pinned to 2add963021)

Solutions

Correct the source bucket name in the path/configuration
Verify the bucket exists and is reachable with the same credentials: gsutil ls gs://<src-bucket>
If the bucket belongs to another project, grant the service account access there

Defensive patterns

Strategy: validation

Validate before calling

// Verify the source bucket exists before cross-bucket copy
GoogleCloudStorageItemInfo info = gcs.getItemInfo(new StorageResourceId(srcBucket));
if (!info.exists()) {
  throw new FileNotFoundException("Source bucket missing: " + srcBucket);
}
gcs.copy(map);

Prevention

Validate gs:// bucket names at configuration load time
Run gsutil ls on both buckets under job credentials as a pre-flight check
Grant cross-project bucket access where needed — invisibility looks like absence

When it happens

Trigger: Calling copy() where source and destination bucket names differ and the SOURCE bucket does not exist or is not visible to the project/credentials — e.g. typo'd bucket, deleted bucket, or a bucket in another project without access (which surfaces as not-found).

Common situations: Typo in the gs:// source URL; bucket deleted between job submission and run; credentials scoped to a different project where the bucket is invisible; distcp configs copied between environments.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4fe9b02b37ff3316. Report an issue: GitHub.