apache/hadoop · error · IOException

Filesystem closed

Error message

Filesystem closed

What it means

DFSClient.checkOpen() guards essentially every client operation: once clientRunning is false (set by DFSClient.close(), which DistributedFileSystem.close() triggers), any subsequent RPC attempt throws IOException('Filesystem closed'). It means the handle is used after its owning FileSystem/DFSClient was closed, often by different code than the failing call.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:490

   */
  int getDatanodeWriteTimeout(int numNodes) {
    final int t = dfsClientConf.getDatanodeSocketWriteTimeout();
    return t > 0? t + HdfsConstants.WRITE_TIMEOUT_EXTENSION*numNodes: 0;
  }

  int getDatanodeReadTimeout(int numNodes) {
    final int t = dfsClientConf.getSocketTimeout();
    return t > 0? HdfsConstants.READ_TIMEOUT_EXTENSION*numNodes + t: 0;
  }

  @VisibleForTesting
  public String getClientName() {
    return clientName;
  }

  void checkOpen() throws IOException {
    if (!clientRunning) {
      throw new IOException("Filesystem closed");
    }
  }

  /** Return the lease renewer instance. The renewer thread won't start
   *  until the first output stream is created. The same instance will
   *  be returned until all output streams are closed.
   */
  public LeaseRenewer getLeaseRenewer() {
    return LeaseRenewer.getInstance(
        namenodeUri != null ? namenodeUri.getAuthority() : "null", ugi, this);
  }

  /** Get a lease and start automatic renewal */
  private void beginFileLease(final String key, final DFSOutputStream out) {
    synchronized (filesBeingWritten) {
      putFileBeingWritten(key, out);
      LeaseRenewer renewer = getLeaseRenewer();
      boolean result = renewer.put(this);

View on GitHub (pinned to 2add963021)

Solutions

1. Pick one ownership model: for FileSystem.get(conf) never call close() (the cache owns it); if you must own closing, create with FileSystem.newInstance(conf) and close exactly once, ideally via try-with-resources.
2. Fix double-ownership: find the component calling close() on the shared instance (thread dumps, audit code paths around cleanup hooks).
3. After UGI relogin/logout cycles, drop references and re-acquire FileSystem handles.
4. As a runtime remedy, catch IOException 'Filesystem closed', discard the reference, get a fresh instance, and retry the operation once.

Example fix

// before
FileSystem fs = FileSystem.get(conf);
try { read(fs); } finally { fs.close(); }
// poisons the shared cached instance; later FileSystem.get(conf) users get
// IOException: Filesystem closed

// after (cached, shared - do not close)
FileSystem fs = FileSystem.get(conf);
read(fs);

// after (owned instance - close is safe)
try (FileSystem fs = FileSystem.newInstance(conf)) {
  read(fs);
}

// no public isOpen(); cheap probe before critical sections
boolean usable;
try { fs.getStatus(); usable = true; }
catch (IOException e) { usable = !"Filesystem closed".equals(e.getMessage()); }
if (!usable) { fs = FileSystem.get(conf); /* fresh cached handle */ }

catch (IOException e) {
  if ("Filesystem closed".equals(e.getMessage())) {
    fs = FileSystem.newInstance(conf); // or FileSystem.get(conf) after cache invalidation
    // retry the operation once on the fresh instance
  } else { throw e; }
}

When it happens

Trigger: Calling fs.open()/read()/listStatus()/getFileInfo() after fs.close(); using a DFSInputStream whose parent FileSystem was closed; one thread closing a shared cached FileSystem obtained via FileSystem.get() while another thread still uses the cached instance; UserGroupInformation relogin/logout flows that call FileSystem.closeAllForUGI() invalidating cached handles.

Common situations: User code closing a cached FileSystem (FileSystem.get returns the shared cached instance; closing it poisons later get() callers in the same JVM); try-with-resources wrapping a cached FileSystem; MR/Spark tasks whose cleanup closes shared FS handles; long-lived services doing keytab relogin cycles.

Related errors

• Stream closed
• Credentials requested after provider list was closed
• GoogleHadoopFileSystem has been closed or not initialized.
• Stream is closed!
• {} doesn't support satisfyStoragePolicy

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca4bd351daf3315e. Report an issue: GitHub.