{"record":{"id":"91df0417786130f5","repo":"apache/hadoop","slug":"bucket-does-not-exist-91df04","errorCode":null,"errorMessage":" Bucket does not exist","messagePattern":" Bucket does not exist","errorType":"exception","errorClass":"UnknownStoreException","httpStatus":404,"severity":"critical","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java","lineNumber":3071,"sourceCode":"    return getStore().headObject(key, changeTracker, changeInvoker, fsHandler, operation);\n  }\n\n  /**\n   * Request bucket metadata.\n   * @return the metadata\n   * @throws UnknownStoreException the bucket is absent\n   * @throws IOException  any other problem talking to S3\n   */\n  @AuditEntryPoint\n  @Retries.RetryTranslated\n  protected HeadBucketResponse getBucketMetadata() throws IOException {\n    final HeadBucketResponse response = trackDurationAndSpan(STORE_EXISTS_PROBE, bucket, null,\n        () -> invoker.retry(\"getBucketMetadata()\", bucket, true, () -> {\n          try {\n            return getS3Client().headBucket(\n                getRequestFactory().newHeadBucketRequestBuilder(bucket).build());\n          } catch (NoSuchBucketException e) {\n            throw new UnknownStoreException(\"s3a://\" + bucket + \"/\", \" Bucket does \" + \"not exist\");\n          }\n        }));\n    return response;\n  }\n\n  /**\n   * Initiate a {@code listObjects} operation, incrementing metrics\n   * in the process.\n   *\n   * Retry policy: retry untranslated.\n   * @param request request to initiate\n   * @param trackerFactory duration tracking\n   * @return the results\n   * @throws IOException if the retry invocation raises one (it shouldn't).\n   */\n  @Retries.RetryRaw\n  protected S3ListResult listObjects(S3ListRequest request,\n      @Nullable final DurationTrackerFactory trackerFactory)","sourceCodeStart":3053,"sourceCodeEnd":3089,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3053-L3089","documentation":"getBucketMetadata() runs headBucket through the retrying invoker; a NoSuchBucketException from S3 is translated to UnknownStoreException('s3a://bucket/', ' Bucket does not exist'). Unlike the initialize-time probe (verifyBucketExists), this fires on later calls that need bucket metadata (e.g. HeaderProcessing when preparing requests), so a bucket that disappears or becomes unreachable after the FileSystem was created surfaces here.","triggerScenarios":"Bucket deleted while a cached S3AFileSystem is still in use; endpoint/region routing changes making headBucket reach a store where the bucket is absent; access-point or FIPS endpoints where the bucket is not visible.","commonSituations":"Long-lived JVMs (Hive Metastore, Spark, NameNode-side tooling) reusing cached FileSystems after bucket deletion; MinIO/Ceph restarted with fresh state; DNS or endpoint reconfiguration mid-session.","solutions":["Recreate the bucket, or close and evict the cached FileSystem so the next FileSystem.get() re-initializes and re-probes","Verify fs.s3a.endpoint and fs.s3a.endpoint.region still point at the owning region","After bucket lifecycle changes in ops tooling, proactively close cached instances (FileSystem.closeAllForUGI / per-FS close)"],"exampleFix":"// before: long-lived reference keeps failing after bucket ops\nFileSystem fs = cache.get();\nfs.rename(src, dst);\n\n// after: on UnknownStoreException, drop and rebuild the client\ntry {\n  fs.rename(src, dst);\n} catch (UnknownStoreException e) {\n  fs.close();\n  fs = FileSystem.newInstance(srcUri, conf);\n}","handlingStrategy":"try-catch","validationCode":"// Cheap liveness probe for long-lived clients\nnew S3AFileSystem() /* not direct */ - use a status call instead:\ntry {\n  fs.getFileStatus(new Path(\"/\"));\n} catch (UnknownStoreException e) {\n  // bucket vanished under a cached FS instance\n}","typeGuard":"static boolean bucketVanished(Throwable t) {\n  return t instanceof org.apache.hadoop.fs.UnknownStoreException;\n}","tryCatchPattern":"Catch UnknownStoreException on operations against long-lived FileSystems: close the broken instance, rebuild with FileSystem.newInstance(uri, conf) (or let the cache re-initialize), and fail the operation with context - do not retry against the dead reference.","preventionTips":["After bucket deletion/recreation events, close cached FileSystems (FileSystem.closeAllForUGI) in the same runbook step","Monitor bucket existence in ops tooling for long-lived services","Keep endpoint/region configuration pinned so headBucket routes to the owning region"],"tags":["s3a","s3","bucket","unknown-store","cache-invalidation"],"backgroundTag":"s3-bucket-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}