apache/hadoop · error · NoAuthWithCOSException

Credentials requested after provider list was closed

Error message

Credentials requested after provider list was closed

What it means

COSCredentialsProviderList is refcounted: share() increments, close() decrements and marks the list closed when the last user releases it. getCredentials() on an already-closed list throws NoAuthWithCOSException("Credentials requested after provider list was closed"). This is a lifecycle bug — some component holds/uses the credential chain beyond the lifetime of the filesystem that owned it.

Source

Thrown at hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/auth/COSCredentialsProviderList.java:92

    if (this.providers.isEmpty()) {
      throw new NoAuthWithCOSException(NO_COS_CREDENTIAL_PROVIDERS);
    }
  }

  public COSCredentialsProviderList share() {
    Preconditions.checkState(!this.closed(), "Provider list is closed");
    this.refCount.incrementAndGet();
    return this;
  }

  public boolean closed() {
    return this.isClosed.get();
  }

  @Override
  public COSCredentials getCredentials() {
    if (this.closed()) {
      throw new NoAuthWithCOSException(CREDENTIALS_REQUESTED_WHEN_CLOSED);
    }

    this.checkNotEmpty();

    if (this.reuseLastProvider && this.lastProvider != null) {
      return this.lastProvider.getCredentials();
    }

    for (COSCredentialsProvider provider : this.providers) {
      COSCredentials credentials = provider.getCredentials();
      if (null != credentials
           && !StringUtils.isNullOrEmpty(credentials.getCOSAccessKeyId())
           && !StringUtils.isNullOrEmpty(credentials.getCOSSecretKey())
           || credentials instanceof AnonymousCOSCredentials) {
        this.lastProvider = provider;
        return credentials;
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

Close all streams and complete all operations before calling FileSystem.close() on the cosn instance
If sharing the filesystem across components, use FileSystem.get() caching so the same refcounted instance is reused instead of hand-managed lifetimes
Do not cache or export the internal COSCredentialsProviderList beyond the filesystem's lifetime
Audit for double-close paths — a second close() can drop the refcount to zero while workers still run

Example fix

// before
FileSystem fs = path.getFileSystem(conf);
FSDataInputStream in = fs.open(path);
fs.close();               // refcount -> 0, list closed
in.read();               // -> Credentials requested after provider list was closed

// after
FileSystem fs = path.getFileSystem(conf);
try (FSDataInputStream in = fs.open(path)) {
  in.read();
}
fs.close();

Defensive patterns

Strategy: validation

Validate before calling

// Guard the credential chain's lifecycle before use
COSCredentialsProviderList list = ...;
if (list.closed()) {
  // re-acquire the filesystem instead of using a dead chain
  fs = FileSystem.get(uri, conf);
}

Try / catch

try {
  fs.open(path);
} catch (NoAuthWithCOSException e) {
  if (String.valueOf(e.getMessage()).contains("provider list was closed")) {
    // lifecycle bug: someone closed the FileSystem while streams were alive
    throw new IllegalStateException("cosn FileSystem used after close", e);
  }
  throw e;
}

Prevention

Close all FSDataInputStream/OutputStream instances before FileSystem.close()
Rely on FileSystem.get() caching rather than hand-managed lifetimes
Never cache the internal provider list beyond the owning filesystem

When it happens

Trigger: Calling getCredentials() (directly or via a store/stream still alive) after CosNFileSystem.close() released the last reference; double-closing a CosNFileSystem while background threads (upload part threads, read streams) still authenticate; caching a provider list across filesystem instances that then closes it.

Common situations: Not closing streams before closing the filesystem; jobs that reuse a cached FileSystem object after the owner task closed it; custom code that grabs the internal provider list and outlives the FS; shutdown ordering bugs in embedded usage.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/49c52ef1d16f8b56. Report an issue: GitHub.