apache/hadoop · error · IOException

Filesystem %s closed

Error message

Filesystem %s closed

What it means

OBSBlockOutputStream.checkOpen() throws IOException('Filesystem <writeOperationHelper.toString(key)> closed') when any write/flush operation is attempted after the owning OBSFileSystem instance was closed (AtomicBoolean 'closed' set). The stream keeps a reference to the filesystem's writeOperationHelper, so once fs.close() runs (e.g. at job teardown) every subsequent write on any still-open stream fails with this message — it is a lifecycle bug in the caller, not an OBS service error.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java:255

  /**
   * Clear the active block.
   */
  private synchronized void clearActiveBlock() {
    if (activeBlock != null) {
      LOG.debug("Clearing active block");
    }
    activeBlock = null;
  }

  /**
   * Check for the filesystem being open.
   *
   * @throws IOException if the filesystem is closed.
   */
  private void checkOpen() throws IOException {
    if (closed.get()) {
      throw new IOException(
          "Filesystem " + writeOperationHelper.toString(key) + " closed");
    }
  }

  /**
   * The flush operation does not trigger an upload; that awaits the next block
   * being full. What it does do is call {@code flush() } on the current block,
   * leaving it to choose how to react.
   *
   * @throws IOException Any IO problem.
   */
  @Override
  public synchronized void flush() throws IOException {
    checkOpen();
    OBSDataBlocks.DataBlock dataBlock = getActiveBlock();
    if (dataBlock != null) {
      dataBlock.flush();
    }

View on GitHub (pinned to 2add963021)

Solutions

  1. Fix ownership: close output streams BEFORE closing the FileSystem (try-with-resources nested correctly), and only close the FS when no writers remain.
  2. For shared FileSystems use FileSystem.closeAllForUGI or reference-counted caches instead of manual fs.close() on a cached instance.
  3. Cancel/join background writer threads before FS teardown (await termination of executors that touch the stream).
  4. In Spark, prefer path-based lazy writes or re-obtain the FileSystem per task instead of caching across task lifecycle boundaries.

Example fix

// before
FSDataOutputStream out = fs.create(path);
executor.submit(() -> { out.write(buf); });
fs.close(); // writer thread now hits 'Filesystem ... closed'

// after
FSDataOutputStream out = fs.create(path);
Future<?> f = executor.submit(() -> { out.write(buf); });
f.get(30, TimeUnit.SECONDS);
out.close();
fs.close(); // streams closed first, writers joined
Defensive patterns

Strategy: validation

Validate before calling

boolean fsUsable = true; // track alongside every cached FileSystem
// guard before each write batch:
if (fsClosed) throw new IllegalStateException("filesystem already closed for " + path);

Try / catch

try {
  out.write(buf, 0, n);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).endsWith(" closed") && e.getMessage().contains("Filesystem")) {
    throw new LifecycleException("Writer outlived OBSFileSystem — close streams before fs.close()", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling fs.close() in a finally block while background threads still write through cached output streams; caching an FSDataOutputStream across tasks and closing the FileSystem between them; Spark executor reuse where a UDF writes lazily after the cluster shuts the FS; Timer/async threads flushing metrics files post-shutdown.

Common situations: Static/shared FileSystem singletons closed by one task while another still writes; shutdown hooks racing appender threads; mapreduce task commit closing the FS before a slow record writer finishes; tests that close fs in @AfterEach while async writers linger.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c57818ce4b7a2a5b. Report an issue: GitHub.