apache/hadoop · error · IOException

{}: FileSystem is closed!

Error message

{}: FileSystem is closed!

What it means

SFTPFileSystem.connect() begins with checkNotClosed() (defined at SFTPFileSystem.java:729), so once close() has run (closed AtomicBoolean set, connection pool shut down), every channel-acquiring operation — open, create, rename, delete, listStatus, getFileStatus, mkdirs — throws IOException formatted as '<uri>: FileSystem is closed!'.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java:731

      super.close();
      if (closed.getAndSet(true)) {
        return;
      }
    } finally {
      if (connectionPool != null) {
        connectionPool.shutdown();
      }
    }
  }

  /**
   * Verify that the input stream is open. Non blocking; this gives
   * the last state of the volatile {@link #closed} field.
   * @throws IOException if the connection is closed.
   */
  private void checkNotClosed() throws IOException {
    if (closed.get()) {
      throw new IOException(uri + ": " + E_FS_CLOSED);
    }
  }

  @VisibleForTesting
  SFTPConnectionPool getConnectionPool() {
    return connectionPool;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Obtain a fresh, uncached instance with FileSystem.newInstance(uri, conf) instead of reusing or re-getting the closed one.
2. Audit for premature close(): close sftp:// filesystems only at application shutdown, and give each instance a single owner responsible for closing.
3. If a framework may close shared cached instances, disable caching (fs.<scheme>.impl.disable.cache=true) or re-acquire the instance per task.

Example fix

// before
FileSystem fs = FileSystem.get(sftpUri, conf);
fs.close();
fs.open(path); // IOException: FileSystem is closed!

// after
FileSystem fs = FileSystem.get(sftpUri, conf);
// ... all work with fs ...
fs.close(); // last statement; re-acquire via newInstance() if needed later

try {
  in = fs.open(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("FileSystem is closed")) {
    fs = FileSystem.newInstance(uri, conf); // fresh, uncached instance
    in = fs.open(path);
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling any SFTPFileSystem data operation after fs.close(); reusing a FileSystem instance obtained from the FileSystem cache after some other component closed that shared cached instance; invoking FileSystem.closeAll() in a cleanup hook while the sftp:// filesystem is still in use.

Common situations: One library calls close() on the cached instance and a later caller gets the same closed object; finally-blocks that close prematurely before a retry loop; test code that closes the filesystem between cases but keeps the reference.

Related errors

• key + ": Stream is closed!"
• Stream closed
• %s: Stream is closed!
• GoogleHadoopFileSystem has been closed or not initialized.
• Stream closed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/def9d98a8c82815f. Report an issue: GitHub.