{"record":{"id":"fffb0ae5df3e274b","repo":"apache/hadoop","slug":"exception-while-get-content-summary","errorCode":null,"errorMessage":"Exception while get content summary","messagePattern":"Exception while get content summary","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java","lineNumber":562,"sourceCode":"        processDirectory(status.getPath(), futures, es,\n            exceptionThrow);\n      }\n    }\n\n    while (!exceptionThrow.get() && !futures.isEmpty()) {\n      Future<ContentSummary> future = futures.poll();\n      try {\n        ContentSummary subSummary = future.get();\n        summary[0] += subSummary.getLength();\n        summary[1] += subSummary.getFileCount();\n        summary[2] += subSummary.getDirectoryCount();\n      } catch (InterruptedException | ExecutionException e) {\n        LOG.error(e.getMessage(), e);\n        throw new RuntimeException(e);\n      }\n    }\n    if (exceptionThrow.get()) {\n      throw new IOException(\n          \"Exception while get content summary\");\n    }\n    return new ContentSummary.Builder()\n        .length(summary[0]).fileCount(summary[1])\n        .directoryCount(summary[2]).build();\n  }\n\n  private void processDirectory(Path p,\n      Queue<Future<ContentSummary>> futures,\n      ExecutorService es, AtomicBoolean exceptionThrow) {\n    futures.add(es.submit(() -> {\n      long[] summary = new long[]{0L, 0L, 0L};\n      try {\n        if (!exceptionThrow.get()) {\n          FileStatus[] statuses = listStatus(p);\n          for (FileStatus status : statuses) {\n            if (status.isFile()) {\n              summary[0] += status.getLen();","sourceCodeStart":544,"sourceCodeEnd":580,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java#L544-L580","documentation":"TypedBytesWritableInput.readType() reads the next type byte and returns the corresponding java.lang.Class<? extends Writable> (BytesWritable, Text, MapWritable, ...). Unknown codes return null at the TypedBytesInput layer, so the switch default that throws RuntimeException 'unknown type' is reached only for enum constants with no Writable mapping — in practice MARKER (255), and theoretically new Type constants after a jar version mismatch. It is the class-probing counterpart of read().","triggerScenarios":"Calling readType() when the stream's next byte is a MARKER (255) sentinel or a type code added by a newer hadoop-streaming version; commonly when peeking types on a raw typed-bytes pipe that uses markers for framing.","commonSituations":"Dynamic readers that peek the upcoming record class; mixing raw typed-bytes pipes (which emit markers) with the Writable-based reader; client/cluster hadoop-streaming version skew.","solutions":["Treat markers before peeking: use TypedBytesInput.readType() (returns null for both EOF and unknown codes) or consume/skip marker bytes before calling the Writable-level readType().","Pin matching hadoop-streaming jar versions on client and cluster (check for duplicate older copies in the job jar's lib/).","Guard with a try/catch and treat the failure as a stream-corruption signal — re-sync by re-reading from the last known-good offset.","For extension codes 50-200, decode manually via TypedBytesInput.readRawBytes instead of expecting readType() to classify them."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"org.apache.hadoop.typedbytes.Type t = rawIn.readType();\nif (t == null || t == org.apache.hadoop.typedbytes.Type.MARKER) {\n  // not a data record: skip or resynchronize instead of calling readType()\n}","typeGuard":null,"tryCatchPattern":"try {\n  Class<? extends Writable> cls = tbIn.readType();\n} catch (RuntimeException e) {\n  if (\"unknown type\".equals(e.getMessage()))\n    throw new IOException(\"unmappable typed-bytes type code\", e);\n  throw e;\n}","preventionTips":["Peek with TypedBytesInput.readType() (null-safe) before the Writable-layer call.","Consume markers explicitly on raw pipes.","Pin one hadoop-streaming version across the pipeline."],"tags":["hadoop","typedbytes","serialization","unknown-type","writable"],"backgroundTag":"deserialization-type-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}