apache/hadoop · error · EntryWriteException

Failed to write manifest data for task {taskId}to local file

Error message

Failed to write manifest data for task {taskId}to local file

What it means

During job commit, LoadManifestsStage streams every manifest's file entries into a local entry sequence file through a bounded queue writer. EntryWriteException('Failed to write manifest data for task <id>to local file') is thrown when EntryWriter.enqueue() returns false - the queue is full and the writer thread is not draining it, classically because the local disk holding the entry sequence file is full or the writer thread has already failed and stopped consuming. The job commit aborts at the AM.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/LoadManifestsStage.java:213

    final String attemptID = manifest.getTaskAttemptID();
    LOG.debug("{}: task attempt {} added {} directories",
        getName(), attemptID, created);

    // add to the summary.
    summaryInfo.add(manifest);

    // clear the manifest extra data so if
    // blocked waiting for queue capacity,
    // memory use is reduced.
    manifest.setIOStatistics(null);
    manifest.getExtraData().clear();

    // queue those files.
    final boolean enqueued = entryWriter.enqueue(manifest.getFilesToCommit());
    if (!enqueued) {
      LOG.warn("{}: Failed to write manifest for task {}",
          getName(), attemptID);
      throw new EntryWriteException(attemptID);
    }

  }

  /**
   * Coalesce all directories and clear the entry in the manifest.
   * There's only ever one writer at a time, which it is hoped reduces
   * contention. before the lock is acquired: if there are no new directories,
   * the write lock is never needed.
   * @param manifest manifest to process
   * @return the number of directories created;
   */
  @VisibleForTesting
  int coalesceDirectories(final TaskManifest manifest) {

    // build a list of dirs to create.
    // this scans the map
    final List<DirEntry> toCreate = manifest.getDestDirectories().stream()

View on GitHub (pinned to 2add963021)

Solutions

1. Check free space on the AM node's local filesystem (where the entry sequence file lives) and free/extend it.
2. Scan the commit logs above this exception for the writer thread's original error - this exception is often the symptom, not the cause.
3. Reduce file counts (fewer, larger output files; fewer reduce tasks writing many small partitions) so manifests are smaller.
4. After freeing space, rerun the job; commit-time local state is rebuilt from the task manifests.

// before submit: ensure the local filesystem has room for the entry sequence file
File localDir = new File(conf.get("yarn.app.mapreduce.am.stagingDir", "/tmp"));
long usable = localDir.getUsableSpace();
if (usable < MIN_FREE_BYTES) { // scale with expected file count, e.g. 100 bytes/file
  throw new IOException("Insufficient local disk (" + usable + " bytes) for manifest commit");
}

try {
  boolean ok = job.waitForCompletion(true);
} catch (IOException e) {
  if (e.getClass().getSimpleName().equals("EntryWriteException")) {
    // local-side failure at commit: free local disk, reduce output file count, rerun
  }
}

When it happens

Trigger: entryWriter.enqueue(manifest.getFilesToCommit()) returning false during LoadManifestsStage: local scratch disk exhausted where the entry sequence file is written, the writer thread terminated after an earlier IO error, or a task with an enormous file list flooding the bounded queue after the writer stalled.

Common situations: Jobs producing millions of small files so the local entry sequence file grows huge; small or shared local disks on the AM node; yarn.nodemanager.local.dirs filling up; earlier local write errors visible higher in the logs before this failure.

Related errors

• Not a valid manifest file; file status = {status}
• Expected a file renamed from {sourcePath}; found {destStatus
• Disk Check failed!
• Failed to copy {srcReplica} block file to {dstFile}
• Negative length of the file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/98640ee15593dbbe. Report an issue: GitHub.