{"record":{"id":"7a5507659d563bd1","repo":"apache/hadoop","slug":"truncate-is-not-supported-by-checksumfilesystem","errorCode":null,"errorMessage":"Truncate is not supported by ChecksumFileSystem","messagePattern":"Truncate is not supported by ChecksumFileSystem","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java","lineNumber":648,"sourceCode":"      fs = this;\n      in = new ChecksumFSInputChecker(this, f, bufferSize);\n    } else {\n      fs = getRawFileSystem();\n      in = fs.open(f, bufferSize);\n    }\n    return new FSDataBoundedInputStream(fs, f, in);\n  }\n\n  @Override\n  public FSDataOutputStream append(Path f, int bufferSize,\n      Progressable progress) throws IOException {\n    throw new UnsupportedOperationException(\"Append is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public boolean truncate(Path f, long newLength) throws IOException {\n    throw new UnsupportedOperationException(\"Truncate is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public void concat(final Path f, final Path[] psrcs) throws IOException {\n    throw new UnsupportedOperationException(\"Concat is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  /**\n   * Calculated the length of the checksum file in bytes.\n   * @param size the length of the data file in bytes\n   * @param bytesPerSum the number of bytes in a checksum block\n   * @return the number of bytes in the checksum file\n   */\n  public static long getChecksumLength(long size, int bytesPerSum) {\n    //the checksum length is equal to size passed divided by bytesPerSum +\n    //bytes written in the beginning of the checksum file.","sourceCodeStart":630,"sourceCodeEnd":666,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L630-L666","documentation":"ChecksumFileSystem (the wrapper behind LocalFileSystem) stores per-chunk CRCs in a sidecar .crc file. Truncating the data file would invalidate every checksum chunk after the cut point, and the wrapper cannot rewrite the .crc safely, so truncate(Path, long) always throws UnsupportedOperationException. Real truncation must be done on the raw filesystem (RawLocalFileSystem.truncate uses RandomAccessFile.setLength) or by rewriting the file.","triggerScenarios":"Calling FileSystem.truncate(Path, long) on LocalFileSystem or any ChecksumFileSystem subclass, e.g. FileSystem.getLocal(conf).truncate(p, newLen). Anything using truncate to shrink local spill/output files through the checksummed local handle hits this immediately.","commonSituations":"Code tested against HDFS (truncate supported since 2.7) being reused on file:// paths; test harnesses that shrink local fixture files through the FileSystem API; tools migrating from RawLocalFileSystem to LocalFileSystem.","solutions":["Truncate via the raw layer: ((ChecksumFileSystem) fs).getRawFileSystem().truncate(f, newLength). Then delete the sidecar .crc file (same name prefixed with a dot, suffixed .crc) so future checksummed reads do not fail on the stale checksums.","Rewrite instead: stream the first newLength bytes to a temp file, then rename it over the original, letting a fresh .crc be created.","In generic multi-filesystem code, catch UnsupportedOperationException from truncate and fall back to a copy-truncate-rename sequence."],"exampleFix":"// before\nLocalFileSystem lfs = FileSystem.getLocal(conf);\nlfs.truncate(path, 1024); // throws UnsupportedOperationException\n\n// after\nlfs.getRawFileSystem().truncate(path, 1024);\nFile crc = new File(path.toUri().getPath() + \".crc\");\nif (crc.exists()) crc.delete(); // .crc no longer matches truncated file","handlingStrategy":"validation","validationCode":"if (fs instanceof ChecksumFileSystem) {\n  // truncate() unsupported: use getRawFileSystem().truncate + drop .crc, or rewrite\n}","typeGuard":"static boolean canTruncate(FileSystem fs) {\n  return !(fs instanceof ChecksumFileSystem);\n}","tryCatchPattern":"try {\n  fs.truncate(path, newLen);\n} catch (UnsupportedOperationException e) {\n  // fall back: raw truncate or stream-copy first newLen bytes to temp and rename\n}","preventionTips":["Test truncate logic against LocalFileSystem in CI so unsupported paths surface early.","After any raw-layer truncate, remove the stale .crc sidecar.","Prefer rewrite-via-temp-and-rename for portable truncation semantics."],"tags":["hadoop","filesystem","truncate","local-filesystem","checksum-fs","unsupported-operation"],"backgroundTag":"unsupported-filesystem-operation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}