apache/hadoop · error · IOException

One or more threads encountered exception during close. See

Error message

One or more threads encountered exception during close. See prior errors.

What it means

Thrown as IOException from MultipleOutputs.close (MultipleOutputs.java:621). close() shuts every cached RecordWriter in a fixed thread pool (mapreduce.multiple-outputs-close-threads, default 10); each writer's close() runs as a Callable that catches its IOException, logs it ('Error while closing MultipleOutput file') and sets encounteredException. This aggregate message is raised only at the end, meaning at least one underlying writer failed to close — the real cause is in the log lines immediately before the exception.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java:621

          writer.close(context);
        } catch (IOException e) {
          LOG.error("Error while closing MultipleOutput file", e);
          encounteredException.set(true);
        }
        return null;
      });
    }
    try {
      executorService.invokeAll(callableList);
    } catch (InterruptedException e) {
      LOG.warn("Closing is Interrupted");
      Thread.currentThread().interrupt();
    } finally {
      executorService.shutdown();
    }

    if (encounteredException.get()) {
      throw new IOException(
          "One or more threads encountered exception during close. See prior errors.");
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Find the real error: search the task log for 'Error while closing MultipleOutput file' or 'failed unexpectedly' immediately preceding this IOException — fix that root cause (FS availability, lease, codec)
For object stores, verify bucket/credentials and prefer a store-native committer so writer close is commit-safe
Tune the close thread count (mapreduce.multiple-outputs-close-threads) down to 1-2 if a shared FS chokes on parallel closes
If the task was killed near its timeout, raise mapreduce.task.timeout so cleanup is not racing container death

Example fix

// cleanup: keep the aggregate, but surface per-channel failures clearly
// before
protected void cleanup(Context context) throws IOException, InterruptedException {
  mos.close(); // aggregate IOException hides the failing channel
}

// after: log which channel is problematic by closing inspected channels yourself is NOT advised;
// instead capture cause context around the call and check task logs for the per-writer ERROR line
protected void cleanup(Context context) throws IOException, InterruptedException {
  try {
    mos.close();
  } catch (IOException e) {
    LOG.error("MultipleOutputs close failed for task {}", context.getTaskAttemptID(), e);
    throw e;
  }
}

Defensive patterns

Strategy: try-catch

Validate before calling

// before cleanup: verify the output FileSystem is reachable
try {
  FileSystem fs = FileOutputFormat.getOutputPath(
      new JobContextImpl(context.getConfiguration(), context.getJobID())).getFileSystem(context.getConfiguration());
  if (!fs.exists(fs.getWorkingDirectory())) throw new IOException("FS unhealthy at cleanup");
} catch (IOException e) { LOG.warn("FS check before mos.close failed", e); }

Try / catch

try { mos.close(); } catch (IOException agg) { // aggregate: real cause logged earlier as 'Error while closing MultipleOutput file' LOG.error("MultipleOutputs close failed for {} — see prior per-writer errors", context.getTaskAttemptID(), agg); throw agg; }

Prevention

Always grep the lines above this exception for the per-writer ERROR — the aggregate alone says nothing
Keep mapreduce.multiple-outputs-close-threads low when the target FS dislikes parallel closes
Give tasks enough timeout headroom for cleanup of many channels
Call mos.close() exactly once in cleanup()

When it happens

Trigger: mos.close() in Mapper/Reducer cleanup where any channel's RecordWriter.close fails: FileSystem errors flushing/closing part files (NN unreachable, lease recovery, S3A multipart abort), compression codec finish() failures (corrupt stream, native lib unload), or an unexpected Throwable in a close thread caught by the UncaughtExceptionHandler.

Common situations: HDFS lease/NameNode hiccups exactly at task cleanup; object-store committers failing multipart complete; tasks being killed near timeout so cleanup races with container teardown; many channels (large thread pool) overloading the FS at once.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/54d4cb62940ee415. Report an issue: GitHub.