apache/hadoop · error · IOException

Exception while get content summary

Error message

Exception while get content summary

What it means

TypedBytesWritableInput.readType() reads the next type byte and returns the corresponding java.lang.Class<? extends Writable> (BytesWritable, Text, MapWritable, ...). Unknown codes return null at the TypedBytesInput layer, so the switch default that throws RuntimeException 'unknown type' is reached only for enum constants with no Writable mapping — in practice MARKER (255), and theoretically new Type constants after a jar version mismatch. It is the class-probing counterpart of read().

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:562

        processDirectory(status.getPath(), futures, es,
            exceptionThrow);
      }
    }

    while (!exceptionThrow.get() && !futures.isEmpty()) {
      Future<ContentSummary> future = futures.poll();
      try {
        ContentSummary subSummary = future.get();
        summary[0] += subSummary.getLength();
        summary[1] += subSummary.getFileCount();
        summary[2] += subSummary.getDirectoryCount();
      } catch (InterruptedException | ExecutionException e) {
        LOG.error(e.getMessage(), e);
        throw new RuntimeException(e);
      }
    }
    if (exceptionThrow.get()) {
      throw new IOException(
          "Exception while get content summary");
    }
    return new ContentSummary.Builder()
        .length(summary[0]).fileCount(summary[1])
        .directoryCount(summary[2]).build();
  }

  private void processDirectory(Path p,
      Queue<Future<ContentSummary>> futures,
      ExecutorService es, AtomicBoolean exceptionThrow) {
    futures.add(es.submit(() -> {
      long[] summary = new long[]{0L, 0L, 0L};
      try {
        if (!exceptionThrow.get()) {
          FileStatus[] statuses = listStatus(p);
          for (FileStatus status : statuses) {
            if (status.isFile()) {
              summary[0] += status.getLen();

View on GitHub (pinned to 2add963021)

Solutions

1. Treat markers before peeking: use TypedBytesInput.readType() (returns null for both EOF and unknown codes) or consume/skip marker bytes before calling the Writable-level readType().
2. Pin matching hadoop-streaming jar versions on client and cluster (check for duplicate older copies in the job jar's lib/).
3. Guard with a try/catch and treat the failure as a stream-corruption signal — re-sync by re-reading from the last known-good offset.
4. For extension codes 50-200, decode manually via TypedBytesInput.readRawBytes instead of expecting readType() to classify them.

org.apache.hadoop.typedbytes.Type t = rawIn.readType();
if (t == null || t == org.apache.hadoop.typedbytes.Type.MARKER) {
  // not a data record: skip or resynchronize instead of calling readType()
}

try {
  Class<? extends Writable> cls = tbIn.readType();
} catch (RuntimeException e) {
  if ("unknown type".equals(e.getMessage()))
    throw new IOException("unmappable typed-bytes type code", e);
  throw e;
}

When it happens

Trigger: Calling readType() when the stream's next byte is a MARKER (255) sentinel or a type code added by a newer hadoop-streaming version; commonly when peeking types on a raw typed-bytes pipe that uses markers for framing.

Common situations: Dynamic readers that peek the upcoming record class; mixing raw typed-bytes pipes (which emit markers) with the Writable-based reader; client/cluster hadoop-streaming version skew.

Related errors

• parent + " is a file"
• Parent directory doesn't exist: " + parent
• f.toString()
• absolutePath + ": No such file or directory."
• absolutePath + " is a file"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fffb0ae5df3e274b. Report an issue: GitHub.