{"record":{"id":"196390e914247ce5","repo":"apache/hadoop","slug":"append-is-not-supported-by-checksumfilesystem","errorCode":null,"errorMessage":"Append is not supported by ChecksumFileSystem","messagePattern":"Append is not supported by ChecksumFileSystem","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java","lineNumber":642,"sourceCode":"   */\n  @Override\n  public FSDataInputStream open(Path f, int bufferSize) throws IOException {\n    FileSystem fs;\n    InputStream in;\n    if (verifyChecksum) {\n      fs = this;\n      in = new ChecksumFSInputChecker(this, f, bufferSize);\n    } else {\n      fs = getRawFileSystem();\n      in = fs.open(f, bufferSize);\n    }\n    return new FSDataBoundedInputStream(fs, f, in);\n  }\n\n  @Override\n  public FSDataOutputStream append(Path f, int bufferSize,\n      Progressable progress) throws IOException {\n    throw new UnsupportedOperationException(\"Append is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public boolean truncate(Path f, long newLength) throws IOException {\n    throw new UnsupportedOperationException(\"Truncate is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public void concat(final Path f, final Path[] psrcs) throws IOException {\n    throw new UnsupportedOperationException(\"Concat is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  /**\n   * Calculated the length of the checksum file in bytes.\n   * @param size the length of the data file in bytes","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L624-L660","documentation":"ChecksumFileSystem is the checksumming wrapper (used by LocalFileSystem) that maintains a sidecar .crc file for every data file. Because it cannot keep the .crc file consistent when bytes are appended, append() unconditionally throws UnsupportedOperationException instead of delegating to the raw filesystem. Append must go through getRawFileSystem(), which returns RawLocalFileSystem (its append is implemented via FileOutputStream), or the file must be rewritten.","triggerScenarios":"Calling FileSystem.append(Path, int, Progressable) (or append(Path), append(Path, int)) on any implementation backed by ChecksumFileSystem: FileSystem.getLocal(conf).append(p), new LocalFileSystem().append(p), or code paths that obtain a LocalFileSystem for file:// URIs (e.g., FsShell copying to a local path with -append,_distcp into file://). The throw is unconditional: any append attempt on this wrapper fails.","commonSituations":"Porting jobs from HDFS (which supports append) to local file:// paths in unit tests or standalone tools; scripts that assumed the generic FileSystem.append contract is universal; upgrading code that previously wrote to RawLocalFileSystem directly and now resolves to the checksummed LocalFileSystem.","solutions":["Rewrite the file instead of appending: read the existing content, open with create(..., overwrite=true), and write old data plus the new bytes. This keeps the .crc file consistent.","If append is mandatory, bypass the checksum layer: ((ChecksumFileSystem) fs).getRawFileSystem().append(f, bufferSize, progress). Note the .crc file becomes stale for the appended tail, so delete the sidecar .crc file or call setVerifyChecksum(false) on readers, or later reads can throw ChecksumException.","Catch UnsupportedOperationException and fall back to copy-and-replace semantics in generic code that runs against multiple filesystem implementations."],"exampleFix":"// before\nLocalFileSystem lfs = FileSystem.getLocal(conf);\ntry (FSDataOutputStream out = lfs.append(path, 4096)) { // throws UnsupportedOperationException\n  out.write(bytes);\n}\n\n// after: append on the raw fs and drop the now-stale checksum sidecar\ntry (FSDataOutputStream out = lfs.getRawFileSystem().append(path, 4096)) {\n  out.write(bytes);\n}\nFile crc = new File(path.toUri().getPath() + \".crc\");\nif (crc.exists()) crc.delete(); // regenerate on next create, or reads will fail checksum","handlingStrategy":"validation","validationCode":"FileSystem fs = ...;\nif (fs instanceof ChecksumFileSystem) {\n  // append() will throw UnsupportedOperationException; plan rewrite or raw-fs append\n}","typeGuard":"static boolean canAppend(FileSystem fs) {\n  return !(fs instanceof ChecksumFileSystem); // LocalFileSystem et al. refuse append\n}","tryCatchPattern":"try {\n  out = fs.append(path, bufferSize);\n} catch (UnsupportedOperationException e) {\n  // rewrite file or use ((ChecksumFileSystem) fs).getRawFileSystem().append(...)\n}","preventionTips":["Default to copy-and-rewrite when targeting file://; treat append as an HDFS/raw-local capability only.","If you bypass to the raw fs, delete the sidecar .crc so later checksummed reads do not fail.","In portable code, probe with instanceof ChecksumFileSystem before choosing the append path."],"tags":["hadoop","filesystem","append","local-filesystem","checksum-fs","unsupported-operation"],"backgroundTag":"unsupported-filesystem-operation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}