apache/hadoop · critical · UnknownStoreException

Bucket does not exist

Error message

 Bucket does not exist

What it means

getBucketMetadata() runs headBucket through the retrying invoker; a NoSuchBucketException from S3 is translated to UnknownStoreException('s3a://bucket/', ' Bucket does not exist'). Unlike the initialize-time probe (verifyBucketExists), this fires on later calls that need bucket metadata (e.g. HeaderProcessing when preparing requests), so a bucket that disappears or becomes unreachable after the FileSystem was created surfaces here.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:3071

    return getStore().headObject(key, changeTracker, changeInvoker, fsHandler, operation);
  }

  /**
   * Request bucket metadata.
   * @return the metadata
   * @throws UnknownStoreException the bucket is absent
   * @throws IOException  any other problem talking to S3
   */
  @AuditEntryPoint
  @Retries.RetryTranslated
  protected HeadBucketResponse getBucketMetadata() throws IOException {
    final HeadBucketResponse response = trackDurationAndSpan(STORE_EXISTS_PROBE, bucket, null,
        () -> invoker.retry("getBucketMetadata()", bucket, true, () -> {
          try {
            return getS3Client().headBucket(
                getRequestFactory().newHeadBucketRequestBuilder(bucket).build());
          } catch (NoSuchBucketException e) {
            throw new UnknownStoreException("s3a://" + bucket + "/", " Bucket does " + "not exist");
          }
        }));
    return response;
  }

  /**
   * Initiate a {@code listObjects} operation, incrementing metrics
   * in the process.
   *
   * Retry policy: retry untranslated.
   * @param request request to initiate
   * @param trackerFactory duration tracking
   * @return the results
   * @throws IOException if the retry invocation raises one (it shouldn't).
   */
  @Retries.RetryRaw
  protected S3ListResult listObjects(S3ListRequest request,
      @Nullable final DurationTrackerFactory trackerFactory)

View on GitHub (pinned to 2add963021)

Solutions

1. Recreate the bucket, or close and evict the cached FileSystem so the next FileSystem.get() re-initializes and re-probes
2. Verify fs.s3a.endpoint and fs.s3a.endpoint.region still point at the owning region
3. After bucket lifecycle changes in ops tooling, proactively close cached instances (FileSystem.closeAllForUGI / per-FS close)

Example fix

// before: long-lived reference keeps failing after bucket ops
FileSystem fs = cache.get();
fs.rename(src, dst);

// after: on UnknownStoreException, drop and rebuild the client
try {
  fs.rename(src, dst);
} catch (UnknownStoreException e) {
  fs.close();
  fs = FileSystem.newInstance(srcUri, conf);
}

// Cheap liveness probe for long-lived clients
new S3AFileSystem() /* not direct */ - use a status call instead:
try {
  fs.getFileStatus(new Path("/"));
} catch (UnknownStoreException e) {
  // bucket vanished under a cached FS instance
}

static boolean bucketVanished(Throwable t) {
  return t instanceof org.apache.hadoop.fs.UnknownStoreException;
}

Catch UnknownStoreException on operations against long-lived FileSystems: close the broken instance, rebuild with FileSystem.newInstance(uri, conf) (or let the cache re-initialize), and fail the operation with context - do not retry against the dead reference.

When it happens

Trigger: Bucket deleted while a cached S3AFileSystem is still in use; endpoint/region routing changes making headBucket reach a store where the bucket is absent; access-point or FIPS endpoints where the bucket is not visible.

Common situations: Long-lived JVMs (Hive Metastore, Spark, NameNode-side tooling) reusing cached FileSystems after bucket deletion; MinIO/Ceph restarted with fresh state; DNS or endpoint reconfiguration mid-session.

Related errors

• Bucket does not exist. Accessing with fs.s3a.endpoint set t
• 42
• 46
• Wrong bucket: %s, in path: %s, expected bucket: %s
• GCS bucket name cannot be empty.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/91df0417786130f5. Report an issue: GitHub.