apache/hadoop · error · UnsupportedOperationException

Truncate is not supported by ChecksumFileSystem

Error message

Truncate is not supported by ChecksumFileSystem

What it means

ChecksumFileSystem (the wrapper behind LocalFileSystem) stores per-chunk CRCs in a sidecar .crc file. Truncating the data file would invalidate every checksum chunk after the cut point, and the wrapper cannot rewrite the .crc safely, so truncate(Path, long) always throws UnsupportedOperationException. Real truncation must be done on the raw filesystem (RawLocalFileSystem.truncate uses RandomAccessFile.setLength) or by rewriting the file.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:648

      fs = this;
      in = new ChecksumFSInputChecker(this, f, bufferSize);
    } else {
      fs = getRawFileSystem();
      in = fs.open(f, bufferSize);
    }
    return new FSDataBoundedInputStream(fs, f, in);
  }

  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public boolean truncate(Path f, long newLength) throws IOException {
    throw new UnsupportedOperationException("Truncate is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public void concat(final Path f, final Path[] psrcs) throws IOException {
    throw new UnsupportedOperationException("Concat is not supported "
        + "by ChecksumFileSystem");
  }

  /**
   * Calculated the length of the checksum file in bytes.
   * @param size the length of the data file in bytes
   * @param bytesPerSum the number of bytes in a checksum block
   * @return the number of bytes in the checksum file
   */
  public static long getChecksumLength(long size, int bytesPerSum) {
    //the checksum length is equal to size passed divided by bytesPerSum +
    //bytes written in the beginning of the checksum file.

View on GitHub (pinned to 2add963021)

Solutions

  1. Truncate via the raw layer: ((ChecksumFileSystem) fs).getRawFileSystem().truncate(f, newLength). Then delete the sidecar .crc file (same name prefixed with a dot, suffixed .crc) so future checksummed reads do not fail on the stale checksums.
  2. Rewrite instead: stream the first newLength bytes to a temp file, then rename it over the original, letting a fresh .crc be created.
  3. In generic multi-filesystem code, catch UnsupportedOperationException from truncate and fall back to a copy-truncate-rename sequence.

Example fix

// before
LocalFileSystem lfs = FileSystem.getLocal(conf);
lfs.truncate(path, 1024); // throws UnsupportedOperationException

// after
lfs.getRawFileSystem().truncate(path, 1024);
File crc = new File(path.toUri().getPath() + ".crc");
if (crc.exists()) crc.delete(); // .crc no longer matches truncated file
Defensive patterns

Strategy: validation

Validate before calling

if (fs instanceof ChecksumFileSystem) {
  // truncate() unsupported: use getRawFileSystem().truncate + drop .crc, or rewrite
}

Type guard

static boolean canTruncate(FileSystem fs) {
  return !(fs instanceof ChecksumFileSystem);
}

Try / catch

try {
  fs.truncate(path, newLen);
} catch (UnsupportedOperationException e) {
  // fall back: raw truncate or stream-copy first newLen bytes to temp and rename
}

Prevention

When it happens

Trigger: Calling FileSystem.truncate(Path, long) on LocalFileSystem or any ChecksumFileSystem subclass, e.g. FileSystem.getLocal(conf).truncate(p, newLen). Anything using truncate to shrink local spill/output files through the checksummed local handle hits this immediately.

Common situations: Code tested against HDFS (truncate supported since 2.7) being reused on file:// paths; test harnesses that shrink local fixture files through the FileSystem API; tools migrating from RawLocalFileSystem to LocalFileSystem.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7a5507659d563bd1. Report an issue: GitHub.