apache/hadoop · error · UnsupportedOperationException

Append is not supported by ChecksumFileSystem

Error message

Append is not supported by ChecksumFileSystem

What it means

ChecksumFileSystem is the checksumming wrapper (used by LocalFileSystem) that maintains a sidecar .crc file for every data file. Because it cannot keep the .crc file consistent when bytes are appended, append() unconditionally throws UnsupportedOperationException instead of delegating to the raw filesystem. Append must go through getRawFileSystem(), which returns RawLocalFileSystem (its append is implemented via FileOutputStream), or the file must be rewritten.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:642

   */
  @Override
  public FSDataInputStream open(Path f, int bufferSize) throws IOException {
    FileSystem fs;
    InputStream in;
    if (verifyChecksum) {
      fs = this;
      in = new ChecksumFSInputChecker(this, f, bufferSize);
    } else {
      fs = getRawFileSystem();
      in = fs.open(f, bufferSize);
    }
    return new FSDataBoundedInputStream(fs, f, in);
  }

  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public boolean truncate(Path f, long newLength) throws IOException {
    throw new UnsupportedOperationException("Truncate is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public void concat(final Path f, final Path[] psrcs) throws IOException {
    throw new UnsupportedOperationException("Concat is not supported "
        + "by ChecksumFileSystem");
  }

  /**
   * Calculated the length of the checksum file in bytes.
   * @param size the length of the data file in bytes

View on GitHub (pinned to 2add963021)

Solutions

  1. Rewrite the file instead of appending: read the existing content, open with create(..., overwrite=true), and write old data plus the new bytes. This keeps the .crc file consistent.
  2. If append is mandatory, bypass the checksum layer: ((ChecksumFileSystem) fs).getRawFileSystem().append(f, bufferSize, progress). Note the .crc file becomes stale for the appended tail, so delete the sidecar .crc file or call setVerifyChecksum(false) on readers, or later reads can throw ChecksumException.
  3. Catch UnsupportedOperationException and fall back to copy-and-replace semantics in generic code that runs against multiple filesystem implementations.

Example fix

// before
LocalFileSystem lfs = FileSystem.getLocal(conf);
try (FSDataOutputStream out = lfs.append(path, 4096)) { // throws UnsupportedOperationException
  out.write(bytes);
}

// after: append on the raw fs and drop the now-stale checksum sidecar
try (FSDataOutputStream out = lfs.getRawFileSystem().append(path, 4096)) {
  out.write(bytes);
}
File crc = new File(path.toUri().getPath() + ".crc");
if (crc.exists()) crc.delete(); // regenerate on next create, or reads will fail checksum
Defensive patterns

Strategy: validation

Validate before calling

FileSystem fs = ...;
if (fs instanceof ChecksumFileSystem) {
  // append() will throw UnsupportedOperationException; plan rewrite or raw-fs append
}

Type guard

static boolean canAppend(FileSystem fs) {
  return !(fs instanceof ChecksumFileSystem); // LocalFileSystem et al. refuse append
}

Try / catch

try {
  out = fs.append(path, bufferSize);
} catch (UnsupportedOperationException e) {
  // rewrite file or use ((ChecksumFileSystem) fs).getRawFileSystem().append(...)
}

Prevention

When it happens

Trigger: Calling FileSystem.append(Path, int, Progressable) (or append(Path), append(Path, int)) on any implementation backed by ChecksumFileSystem: FileSystem.getLocal(conf).append(p), new LocalFileSystem().append(p), or code paths that obtain a LocalFileSystem for file:// URIs (e.g., FsShell copying to a local path with -append,_distcp into file://). The throw is unconditional: any append attempt on this wrapper fails.

Common situations: Porting jobs from HDFS (which supports append) to local file:// paths in unit tests or standalone tools; scripts that assumed the generic FileSystem.append contract is universal; upgrading code that previously wrote to RawLocalFileSystem directly and now resolves to the checksummed LocalFileSystem.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/196390e914247ce5. Report an issue: GitHub.