{"record":{"id":"19b6e1629e802edf","repo":"apache/hadoop","slug":"not-a-valid-manifest-file-file-status-status","errorCode":null,"errorMessage":"Not a valid manifest file; file status = {status}","messagePattern":"Not a valid manifest file; file status = (.+?)","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/LoadManifestsStage.java","lineNumber":259,"sourceCode":"          directories.putIfAbsent(entry.getDir(), entry);\n        });\n      }\n    }\n    return toCreate.size();\n  }\n\n  /**\n   * Precommit preparation of a single manifest file.\n   * To reduce the memory foot print, the IOStatistics and\n   * extra data of each manifest is cleared.\n   * @param status status of file.\n   * @return number of files.\n   * @throws IOException IO Failure.\n   */\n  private TaskManifest fetchTaskManifest(FileStatus status)\n      throws IOException {\n    if (status.getLen() == 0 || !status.isFile()) {\n      throw new PathIOException(status.getPath().toString(),\n          \"Not a valid manifest file; file status = \" + status);\n    }\n    // load the manifest, which includes validation.\n    final TaskManifest manifest = loadManifest(status);\n    final String id = manifest.getTaskAttemptID();\n    final int filecount = manifest.getFilesToCommit().size();\n    final long size = manifest.getTotalFileSize();\n    LOG.info(\"{}: Task Attempt {} file {}: File count: {}; data size={}\",\n        getName(), id, status.getPath(), filecount, size);\n\n    // record file size for tracking of memory consumption, work etc.\n    final IOStatisticsStore iostats = getIOStatistics();\n    iostats.addSample(COMMITTER_TASK_MANIFEST_FILE_SIZE, status.getLen());\n    iostats.addSample(COMMITTER_TASK_FILE_COUNT_MEAN, filecount);\n    iostats.addSample(COMMITTER_TASK_DIRECTORY_COUNT_MEAN,\n        manifest.getDestDirectories().size());\n    return manifest;\n  }","sourceCodeStart":241,"sourceCodeEnd":277,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/LoadManifestsStage.java#L241-L277","documentation":"While loading task manifests at job commit, fetchTaskManifest() sanity-checks each manifest file's FileStatus and throws PathIOException('Not a valid manifest file; file status = <status>') when the file is zero-length or is not a regular file. Since manifests are written via temp-file-plus-rename, a 0-byte or non-file manifest means the write was interrupted, the FS corrupted it, or something foreign sits in the manifests directory.","triggerScenarios":"A path under the job's task-manifests directory whose status has getLen()==0 or isFile()==false: truncated manifest from a killed/crashed task attempt that still got listed, a directory placed where a manifest file belongs, or FS-level corruption.","commonSituations":"Task killed exactly during manifest publication; external processes or users writing into the committer's manifest directory; disk/FS corruption on the store; mixing committer versions that changed manifest write behavior.","solutions":["Inspect the path and status in the message: a 0-length file points to an interrupted write; a directory points to external interference.","Remove the offending attempt's stale manifest directory and any partial state, then rerun the job.","Verify nothing writes into the committer's directory tree (manifests live under the output path's committer-managed area).","If it recurs, check filesystem health and task-kill logs around the task attempt id shown."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// optional pre-commit audit: every listed manifest must be a non-empty regular file\nfor (FileStatus s : fs.listStatus(manifestsDir)) {\n  if (s.getLen() == 0 || !s.isFile()) {\n    throw new IOException(\"Invalid manifest entry \" + s.getPath() + \" - investigate before commit\");\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (PathIOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Not a valid manifest file\")) {\n    // corrupt/stale manifest: remove that attempt's state and rerun; retrying cannot fix bad bytes\n  }\n}","preventionTips":["Keep external writers out of the committer's manifest directories.","Investigate task kills around manifest publication - they are the usual source of zero-byte manifests.","Verify filesystem health if manifests keep arriving truncated."],"tags":["hadoop","manifest-committer","data-corruption","job-commit","validation"],"backgroundTag":"corrupt-manifest-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}