{"record":{"id":"54d4cb62940ee415","repo":"apache/hadoop","slug":"one-or-more-threads-encountered-exception-during-c-54d4cb","errorCode":null,"errorMessage":"One or more threads encountered exception during close. See prior errors.","messagePattern":"One or more threads encountered exception during close\\. See prior errors\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java","lineNumber":621,"sourceCode":"          writer.close(context);\n        } catch (IOException e) {\n          LOG.error(\"Error while closing MultipleOutput file\", e);\n          encounteredException.set(true);\n        }\n        return null;\n      });\n    }\n    try {\n      executorService.invokeAll(callableList);\n    } catch (InterruptedException e) {\n      LOG.warn(\"Closing is Interrupted\");\n      Thread.currentThread().interrupt();\n    } finally {\n      executorService.shutdown();\n    }\n\n    if (encounteredException.get()) {\n      throw new IOException(\n          \"One or more threads encountered exception during close. See prior errors.\");\n    }\n  }\n}\n\n","sourceCodeStart":603,"sourceCodeEnd":627,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java#L603-L627","documentation":"Thrown as IOException from MultipleOutputs.close (MultipleOutputs.java:621). close() shuts every cached RecordWriter in a fixed thread pool (mapreduce.multiple-outputs-close-threads, default 10); each writer's close() runs as a Callable that catches its IOException, logs it ('Error while closing MultipleOutput file') and sets encounteredException. This aggregate message is raised only at the end, meaning at least one underlying writer failed to close — the real cause is in the log lines immediately before the exception.","triggerScenarios":"mos.close() in Mapper/Reducer cleanup where any channel's RecordWriter.close fails: FileSystem errors flushing/closing part files (NN unreachable, lease recovery, S3A multipart abort), compression codec finish() failures (corrupt stream, native lib unload), or an unexpected Throwable in a close thread caught by the UncaughtExceptionHandler.","commonSituations":"HDFS lease/NameNode hiccups exactly at task cleanup; object-store committers failing multipart complete; tasks being killed near timeout so cleanup races with container teardown; many channels (large thread pool) overloading the FS at once.","solutions":["Find the real error: search the task log for 'Error while closing MultipleOutput file' or 'failed unexpectedly' immediately preceding this IOException — fix that root cause (FS availability, lease, codec)","For object stores, verify bucket/credentials and prefer a store-native committer so writer close is commit-safe","Tune the close thread count (mapreduce.multiple-outputs-close-threads) down to 1-2 if a shared FS chokes on parallel closes","If the task was killed near its timeout, raise mapreduce.task.timeout so cleanup is not racing container death"],"exampleFix":"// cleanup: keep the aggregate, but surface per-channel failures clearly\n// before\nprotected void cleanup(Context context) throws IOException, InterruptedException {\n  mos.close(); // aggregate IOException hides the failing channel\n}\n\n// after: log which channel is problematic by closing inspected channels yourself is NOT advised;\n// instead capture cause context around the call and check task logs for the per-writer ERROR line\nprotected void cleanup(Context context) throws IOException, InterruptedException {\n  try {\n    mos.close();\n  } catch (IOException e) {\n    LOG.error(\"MultipleOutputs close failed for task {}\", context.getTaskAttemptID(), e);\n    throw e;\n  }\n}","handlingStrategy":"try-catch","validationCode":"// before cleanup: verify the output FileSystem is reachable\ntry {\n  FileSystem fs = FileOutputFormat.getOutputPath(\n      new JobContextImpl(context.getConfiguration(), context.getJobID())).getFileSystem(context.getConfiguration());\n  if (!fs.exists(fs.getWorkingDirectory())) throw new IOException(\"FS unhealthy at cleanup\");\n} catch (IOException e) { LOG.warn(\"FS check before mos.close failed\", e); }","typeGuard":null,"tryCatchPattern":"try { mos.close(); } catch (IOException agg) { // aggregate: real cause logged earlier as 'Error while closing MultipleOutput file' LOG.error(\"MultipleOutputs close failed for {} — see prior per-writer errors\", context.getTaskAttemptID(), agg); throw agg; }","preventionTips":["Always grep the lines above this exception for the per-writer ERROR — the aggregate alone says nothing","Keep mapreduce.multiple-outputs-close-threads low when the target FS dislikes parallel closes","Give tasks enough timeout headroom for cleanup of many channels","Call mos.close() exactly once in cleanup()"],"tags":["hadoop","mapreduce","multiple-outputs","close","cleanup","concurrency","aggregate-error"],"backgroundTag":"record-writer-close-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}