{"record":{"id":"0ab5774d78e8fdaf","repo":"apache/hadoop","slug":"checksum-file-not-a-length-multiple-of-checksum-si","errorCode":null,"errorMessage":"Checksum file not a length multiple of checksum size in {} at {} checksumpos: {} sumLenread: {}","messagePattern":"Checksum file not a length multiple of checksum size in (.+?) at (.+?) checksumpos: (.+?) sumLenread: (.+?)","errorType":"exception","errorClass":"ChecksumException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java","lineNumber":298,"sourceCode":"        byte[] checksum) throws IOException {\n\n      boolean eof = false;\n      if (needChecksum()) {\n        assert checksum != null; // we have a checksum buffer\n        assert checksum.length % CHECKSUM_SIZE == 0; // it is sane length\n        assert len >= bytesPerSum; // we must read at least one chunk\n\n        final int checksumsToRead = Math.min(\n          len/bytesPerSum, // number of checksums based on len to read\n          checksum.length / CHECKSUM_SIZE); // size of checksum buffer\n        long checksumPos = getChecksumFilePos(pos);\n        if(checksumPos != sums.getPos()) {\n          sums.seek(checksumPos);\n        }\n\n        int sumLenRead = sums.read(checksum, 0, CHECKSUM_SIZE * checksumsToRead);\n        if (sumLenRead >= 0 && sumLenRead % CHECKSUM_SIZE != 0) {\n          throw new ChecksumException(\n            \"Checksum file not a length multiple of checksum size \" +\n            \"in \" + file + \" at \" + pos + \" checksumpos: \" + checksumPos +\n            \" sumLenread: \" + sumLenRead,\n            pos);\n        }\n        if (sumLenRead <= 0) { // we're at the end of the file\n          eof = true;\n        } else {\n          // Adjust amount of data to read based on how many checksum chunks we read\n          len = Math.min(len, bytesPerSum * (sumLenRead / CHECKSUM_SIZE));\n        }\n      }\n      if(pos != datas.getPos()) {\n        datas.seek(pos);\n      }\n      int nread = readFully(datas, buf, offset, len);\n      if (eof && nread > 0) {\n        throw new ChecksumException(\"Checksum error: \"+file+\" at \"+pos, pos);","sourceCodeStart":280,"sourceCodeEnd":316,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L280-L316","documentation":"LocalFileSystem is ChecksumFileSystem over RawLocalFileSystem: it keeps a sibling .<name>.crc file with an 8-byte header (version + bytesPerChecksum, default 512) followed by 4-byte CRC32 records (CHECKSUM_SIZE=4). During reads it loads checksum records; if the number of bytes read from the .crc is not a multiple of 4 it throws ChecksumException \"Checksum file not a length multiple of checksum size...\". The .crc file itself is malformed - typically truncated or partially copied - rather than the data being wrong.","triggerScenarios":"Reading through LocalFileSystem a file whose .<name>.crc was truncated by an interrupted cp/rsync/scp (hidden dotfiles get copied partially), a disk-full crash during local write, or a stale .crc left next to a regenerated data file. Note the constructor only ignores a missing/unreadable crc at open time; a crc whose header parses but whose body is misaligned fails here mid-read.","commonSituations":"Staging/spill directories copied between hosts include the hidden .crc files and one arrives truncated; a job writing locally was killed, leaving a half-written checksum file that later reads trip over.","solutions":["Delete or rename the matching .<name>.crc file - LocalFileSystem treats a missing crc as \"no checksums\" and reads proceed (integrity is then unchecked).","Better: re-copy both the data file and its .crc from the source so checksum coverage is restored.","If you must read immediately, bypass verification with fs.setVerifyChecksum(false) (or `hadoop fs -cat -ignoreCrc`), or configure fs.file.impl to RawLocalFileSystem.","After any bypass, independently verify data integrity (e.g. compare MD5 against the source) since corruption detection is gone."],"exampleFix":"// before: read fails mid-file with ChecksumException\n\n// after: drop the malformed checksum file, then re-read\nPath crc = new Path(f.getParent(), \".\" + f.getName() + \".crc\");\nlocalFs.delete(crc, false);\ntry (FSDataInputStream in = localFs.open(f)) {\n  in.readFully(buf);\n}","handlingStrategy":"fallback","validationCode":"// pre-flight: a .crc whose body is not 4-byte aligned (after the 8-byte header) will fail mid-read\nPath crc = new Path(f.getParent(), \".\" + f.getName() + \".crc\");\nif (localFs.exists(crc)) {\n  long n = localFs.getFileStatus(crc).getLen();\n  if ((n - 8) % 4 != 0) {\n    localFs.delete(crc, false); // drop malformed crc; reads proceed without checksums\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  localFs.verifyChecksum? 0 : 0; // no-op; see fallback below\n} finally { }","preventionTips":["Copy staging directories with tools that preserve or safely skip hidden dotfiles (.name.crc)","Prefer a quick alignment check on (crcLength - 8) % 4 before long local reads","If checksums are not wanted, configure fs.file.impl=RawLocalFileSystem up front instead of failing mid-read"],"tags":["hadoop","filesystem","checksum","local-filesystem","data-corruption"],"backgroundTag":"checksum-file-corrupt","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}