apache/hadoop · error · IOException

Unknown BlockChecksumType: {}

Error message

Unknown BlockChecksumType: {}

What it means

The switch that dispatches on the requested BlockChecksumType (MD5CRC vs COMPOSITE_CRC) has no other cases, so any other value reaches the default and throws 'Unknown BlockChecksumType'. The public enum only defines those two constants, so through configuration this is unreachable; it fires only when getBlockChecksumType() returns something from a mismatched/newer enum class — i.e., mixed Hadoop jar versions on the classpath or forked code.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/FileChecksumHelper.java:461

          blockChecksumForDebug = md5.toString();
        }
        break;
      case COMPOSITE_CRC:
        BlockChecksumType returnedType = PBHelperClient.convert(
            checksumData.getBlockChecksumOptions().getBlockChecksumType());
        if (returnedType != BlockChecksumType.COMPOSITE_CRC) {
          throw new IOException(String.format(
              "Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC",
              returnedType));
        }
        byte[] crcBytes = checksumData.getBlockChecksum().toByteArray();
        if (LOG.isDebugEnabled()) {
          blockChecksumForDebug = CrcUtil.toSingleCrcString(crcBytes);
        }
        getBlockChecksumBuf().write(crcBytes);
        break;
      default:
        throw new IOException(
            "Unknown BlockChecksumType: " + getBlockChecksumType());
      }
      return blockChecksumForDebug;
    }
  }

  /**
   * Replicated file checksum computer.
   */
  static class ReplicatedFileChecksumComputer extends FileChecksumComputer {
    private int blockIdx;

    ReplicatedFileChecksumComputer(String src, long length,
                                   LocatedBlocks blockLocations,
                                   ClientProtocol namenode,
                                   DFSClient client,
                                   ChecksumCombineMode combineMode)
        throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Deduplicate Hadoop artifacts: print ChecksumCombineMode/BlockChecksumType .class.getProtectionDomain().getCodeSource() to find the rogue jar
2. Align every org.apache.hadoop dependency to one version via dependencyManagement or shading
3. Restart the JVM after fixing the classpath — stale enum classes from earlier loads can persist

Example fix

// before: unknown type slips through the switch
case COMPOSITE_CRC: ...
default: throw new IOException("Unknown BlockChecksumType: " + getBlockChecksumType());

// after (build-time): single hadoop version
// mvn dependency:tree -Dincludes=org.apache.hadoop  -> exactly one hdfs-client version
<dependencyManagement>
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs-client</artifactId>
    <version>3.3.6</version>
  </dependency>
</dependencyManagement>

// Verify classpath coherence before checksum jobs:
Class<?> a = BlockChecksumType.MD5CRC.getClass();
Class<?> b = FileChecksumHelper.class; // same ProtectionDomain expected
System.out.println(a.getProtectionDomain().getCodeSource());
System.out.println(b.getProtectionDomain().getCodeSource()); // must match

try {
  checksum = client.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage().contains("Unknown BlockChecksumType")) {
    throw new IllegalStateException("Conflicting Hadoop jars on classpath", e);
  }
  throw e;
}

When it happens

Trigger: FileChecksumComputer.blockChecksum() invoked when the BlockChecksumType instance was loaded from a different (newer or forked) jar than the switch code; reflective/test code injecting a foreign constant; shaded-plugin conflicts inside one JVM.

Common situations: Applications embedding multiple Hadoop versions (e.g., Spark bundling 3.x plus a user-supplied 3.3.x hdfs-client); fat jars that fail to shade org.apache.hadoop consistently; custom Hadoop forks adding a checksum type.

Related errors

• Unknown ChecksumCombineMode: {}
• Unexpected HAServiceState:
• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum from the byte array of length
• Could not create DataChecksum of type %d with bytesPerChecks

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/06c60821e57f156b. Report an issue: GitHub.