{"record":{"id":"1ef53fd71a59f17b","repo":"apache/hadoop","slug":"executor-service-closed-before-writes-could-be-com","errorCode":null,"errorMessage":"Executor Service closed before writes could be completed.","messagePattern":"Executor Service closed before writes could be completed\\.","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"critical","filePath":"hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java","lineNumber":726,"sourceCode":"  /**\n   * Force all data in the output stream to be written to Azure storage.\n   * Wait to return until this is complete. Close the access to the stream and\n   * shutdown the upload thread pool.\n   * If the blob was created, its lease will be released.\n   * Any error encountered caught in threads and stored will be rethrown here\n   * after cleanup.\n   */\n  @Override\n  public synchronized void close() throws IOException {\n    if (closed) {\n      return;\n    }\n\n    try {\n      // Check if Executor Service got shutdown before the writes could be\n      // completed.\n      if (hasActiveBlockDataToUpload() && executorService.isShutdown()) {\n        throw new PathIOException(path, \"Executor Service closed before \"\n            + \"writes could be completed.\");\n      }\n      flushInternal(true);\n    } catch (IOException e) {\n      // Problems surface in try-with-resources clauses if\n      // the exception thrown in a close == the one already thrown\n      // -so we wrap any exception with a new one.\n      // See HADOOP-16785\n      throw wrapException(path, e.getMessage(), e);\n    } finally {\n      if (contextEncryptionAdapter != null) {\n        contextEncryptionAdapter.destroy();\n      }\n      if (hasLease()) {\n        lease.free();\n        lease = null;\n      }\n      lastError = new IOException(FSExceptionMessages.STREAM_IS_CLOSED);","sourceCodeStart":708,"sourceCodeEnd":744,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java#L708-L744","documentation":"In AbfsOutputStream.close(), if there is still buffered block data to upload (hasActiveBlockDataToUpload()) but the stream's executorService is already shut down, close() throws PathIOException(\"Executor Service closed before writes could be completed.\"). Any exception from close is then wrapped via wrapException (HADOOP-16785) so it is not swallowed in try-with-resources. The data was accepted by write() but can no longer be flushed — this is data loss at rest, not a clean close.","triggerScenarios":"The AzureBlobFileSystem instance (and its shared thread pools) was closed before the output stream — e.g., fs.close() in an outer try-with-resources while the inner output stream still had buffered data; a previous fatal error shut the executor; shutdown-hook ordering closing the FS before flushing writers.","commonSituations":"try-with-resources nesting where FileSystem is the outer resource and OutputStream the inner one but closed out of order; frameworks closing the cached FileSystem (FileSystem.closeAllForUGI at job end) while writers are still active; application shutdown hooks that close the FS first.","solutions":["Always close output streams (after flush/hflush/hsync) BEFORE closing the FileSystem","Call hflush() or hsync() before entering any teardown path","Fix shutdown ordering: close writers, then the filesystem, then the JVM","If a prior error shut the executor, expect this on close — the fix is preventing the earlier failure"],"exampleFix":"// before: FS closed first, buffered data lost\ntry (FileSystem fs = path.getFileSystem(conf);\n     FSDataOutputStream out = fs.create(path)) {\n  out.write(data);\n}   // fs.close() may precede out's final flush -> PathIOException\n\n// after: explicit order\nFileSystem fs = path.getFileSystem(conf);\ntry (FSDataOutputStream out = fs.create(path)) {\n  out.write(data);\n  out.hflush();\n} finally {\n  fs.close();   // streams already closed safely\n}","handlingStrategy":"validation","validationCode":"// Enforce close order: flush, close streams, THEN close the FileSystem\nout.hflush();\nout.close();          // no exception => no buffered data was abandoned\nfs.close();           // safe: nothing still references its executors","typeGuard":null,"tryCatchPattern":"try {\n  out.close();\n} catch (PathIOException e) {\n  if (e.getMessage().contains(\"Executor Service closed\")) {\n    // buffered data was lost because the FS/thread pools shut down first;\n    // fix close ordering and re-run the write\n  } else throw e;\n}","preventionTips":["Never close the FileSystem while output streams still have buffered data","hflush()/hsync() before any teardown or shutdown-hook runs","Order shutdown hooks: writers first, filesystem last"],"tags":["azure-blob","abfs","stream-lifecycle","executor-service","data-loss","close-ordering","hadoop"],"backgroundTag":"executor-service-shutdown","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}