apache/hadoop · critical · PathIOException

Executor Service closed before writes could be completed.

Error message

Executor Service closed before writes could be completed.

What it means

In AbfsOutputStream.close(), if there is still buffered block data to upload (hasActiveBlockDataToUpload()) but the stream's executorService is already shut down, close() throws PathIOException("Executor Service closed before writes could be completed."). Any exception from close is then wrapped via wrapException (HADOOP-16785) so it is not swallowed in try-with-resources. The data was accepted by write() but can no longer be flushed — this is data loss at rest, not a clean close.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java:726

  /**
   * Force all data in the output stream to be written to Azure storage.
   * Wait to return until this is complete. Close the access to the stream and
   * shutdown the upload thread pool.
   * If the blob was created, its lease will be released.
   * Any error encountered caught in threads and stored will be rethrown here
   * after cleanup.
   */
  @Override
  public synchronized void close() throws IOException {
    if (closed) {
      return;
    }

    try {
      // Check if Executor Service got shutdown before the writes could be
      // completed.
      if (hasActiveBlockDataToUpload() && executorService.isShutdown()) {
        throw new PathIOException(path, "Executor Service closed before "
            + "writes could be completed.");
      }
      flushInternal(true);
    } catch (IOException e) {
      // Problems surface in try-with-resources clauses if
      // the exception thrown in a close == the one already thrown
      // -so we wrap any exception with a new one.
      // See HADOOP-16785
      throw wrapException(path, e.getMessage(), e);
    } finally {
      if (contextEncryptionAdapter != null) {
        contextEncryptionAdapter.destroy();
      }
      if (hasLease()) {
        lease.free();
        lease = null;
      }
      lastError = new IOException(FSExceptionMessages.STREAM_IS_CLOSED);

View on GitHub (pinned to 2add963021)

Solutions

Always close output streams (after flush/hflush/hsync) BEFORE closing the FileSystem
Call hflush() or hsync() before entering any teardown path
Fix shutdown ordering: close writers, then the filesystem, then the JVM
If a prior error shut the executor, expect this on close — the fix is preventing the earlier failure

Example fix

// before: FS closed first, buffered data lost
try (FileSystem fs = path.getFileSystem(conf);
     FSDataOutputStream out = fs.create(path)) {
  out.write(data);
}   // fs.close() may precede out's final flush -> PathIOException

// after: explicit order
FileSystem fs = path.getFileSystem(conf);
try (FSDataOutputStream out = fs.create(path)) {
  out.write(data);
  out.hflush();
} finally {
  fs.close();   // streams already closed safely
}

Defensive patterns

Strategy: validation

Validate before calling

// Enforce close order: flush, close streams, THEN close the FileSystem
out.hflush();
out.close();          // no exception => no buffered data was abandoned
fs.close();           // safe: nothing still references its executors

Try / catch

try {
  out.close();
} catch (PathIOException e) {
  if (e.getMessage().contains("Executor Service closed")) {
    // buffered data was lost because the FS/thread pools shut down first;
    // fix close ordering and re-run the write
  } else throw e;
}

Prevention

Never close the FileSystem while output streams still have buffered data
hflush()/hsync() before any teardown or shutdown-hook runs
Order shutdown hooks: writers first, filesystem last

When it happens

Trigger: The AzureBlobFileSystem instance (and its shared thread pools) was closed before the output stream — e.g., fs.close() in an outer try-with-resources while the inner output stream still had buffered data; a previous fatal error shut the executor; shutdown-hook ordering closing the FS before flushing writers.

Common situations: try-with-resources nesting where FileSystem is the outer resource and OutputStream the inner one but closed out of order; frameworks closing the cached FileSystem (FileSystem.closeAllForUGI at job end) while writers are still active; application shutdown hooks that close the FS first.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1ef53fd71a59f17b. Report an issue: GitHub.