apache/hadoop · error · PathIOException

Failed to read JSON file ${e}

Error message

Failed to read JSON file ${e}

What it means

When the JSON read from an open FileSystem stream cannot be parsed, the Jackson JsonProcessingException is wrapped in PathIOException(path, "Failed to read JSON file <cause>", e). PathIOException identifies the offending path so callers can blame the file rather than the filesystem, and the cause chain keeps the exact Jackson error (line/column or mismatched property).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java:280

   * @throws IOException IO problems
   */
  public T load(FileSystem fs, Path path, @Nullable FileStatus status)
      throws IOException {

    if (status != null && status.getLen() == 0) {
      throw new EOFException("No data in " + path);
    }
    FutureDataInputStreamBuilder builder = fs.openFile(path)
        .opt(FS_OPTION_OPENFILE_READ_POLICY,
            FS_OPTION_OPENFILE_READ_POLICY_WHOLE_FILE);
    if (status != null) {
      builder.withFileStatus(status);
    }
    try (FSDataInputStream dataInputStream =
             awaitFuture(builder.build())) {
      return fromJsonStream(dataInputStream);
    } catch (JsonProcessingException e) {
      throw new PathIOException(path.toString(),
          "Failed to read JSON file " + e, e);
    }
  }

  /**
   * Save to a Hadoop filesystem.
   * @param fs filesystem
   * @param path path
   * @param overwrite should any existing file be overwritten
   * @param instance instance
   * @throws IOException IO exception.
   */
  public void save(FileSystem fs, Path path, T instance,
      boolean overwrite) throws
      IOException {
    writeJsonAsBytes(instance, fs.create(path, overwrite));
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Dump the file (hdfs dfs -cat) and validate it as JSON, comparing its shape with the target class.
2. Restore or regenerate the corrupted file - re-run the writer, or delete it so it is recreated from defaults.
3. Catch PathIOException in bulk-processing loops and quarantine the bad path instead of aborting the whole run.

Example fix

// before
for (Path p : paths) {
  items.add(serializer.load(fs, p, null)); // one bad file aborts everything
}

// after
for (Path p : paths) {
  try {
    items.add(serializer.load(fs, p, null));
  } catch (PathIOException e) {
    LOG.warn("Quarantining unparseable file {}", p, e);
    quarantine(p);
  }
}

try {
  T v = serializer.load(fs, path, status);
} catch (PathIOException e) {
  // e.getPath() names the file; e.getCause() is the Jackson exception
  // with line/column or the mismatched property
}

When it happens

Trigger: load(fs, path, status) on a non-empty file whose bytes are not valid JSON for classType: truncated JSON, a different schema version, or a non-JSON file stored at the path.

Common situations: State file half-written by a crashed process (length > 0 but content truncated); schema drift after an upgrade changed the JSON shape; an error page or log file accidentally written to the expected path.

Related errors

• No data in ${path}
• Mkdirs failed to create directory {dirName}
• Could not rename {oldDir} to {newDir}
• Failed to add {}: You cannot have a rack and a non-rack node
• No data

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1a87cc81b4a1f8ea. Report an issue: GitHub.