{"record":{"id":"73f52c8ea695fbc9","repo":"apache/hadoop","slug":"concat-is-not-supported-by-checksumfilesystem","errorCode":null,"errorMessage":"Concat is not supported by ChecksumFileSystem","messagePattern":"Concat is not supported by ChecksumFileSystem","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java","lineNumber":654,"sourceCode":"    return new FSDataBoundedInputStream(fs, f, in);\n  }\n\n  @Override\n  public FSDataOutputStream append(Path f, int bufferSize,\n      Progressable progress) throws IOException {\n    throw new UnsupportedOperationException(\"Append is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public boolean truncate(Path f, long newLength) throws IOException {\n    throw new UnsupportedOperationException(\"Truncate is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  @Override\n  public void concat(final Path f, final Path[] psrcs) throws IOException {\n    throw new UnsupportedOperationException(\"Concat is not supported \"\n        + \"by ChecksumFileSystem\");\n  }\n\n  /**\n   * Calculated the length of the checksum file in bytes.\n   * @param size the length of the data file in bytes\n   * @param bytesPerSum the number of bytes in a checksum block\n   * @return the number of bytes in the checksum file\n   */\n  public static long getChecksumLength(long size, int bytesPerSum) {\n    //the checksum length is equal to size passed divided by bytesPerSum +\n    //bytes written in the beginning of the checksum file.\n    return ((size + bytesPerSum - 1) / bytesPerSum) * FSInputChecker.CHECKSUM_SIZE +\n             ChecksumFSInputChecker.HEADER_LENGTH;\n  }\n\n  /** This class provides an output stream for a checksummed file.\n   * It generates checksums for data. */","sourceCodeStart":636,"sourceCodeEnd":672,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java#L636-L672","documentation":"ChecksumFileSystem.concat(Path, Path[]) unconditionally throws UnsupportedOperationException. Concat (server-side concatenation of files without copying) only exists on filesystems with block-level semantics like HDFS; a checksumming local wrapper cannot splice the per-chunk .crc files of the sources, so it refuses the operation.","triggerScenarios":"Calling concat(dst, srcs) on LocalFileSystem or any ChecksumFileSystem subclass, typically in code written against DistributedFileSystem.concat and pointed at file:// paths (unit tests, local shuffle merging, distcp between HDFS and file://).","commonSituations":"Test code reusing HDFS concat logic against local mini-clusters/fixtures; utilities that treat FileSystem.concat as universally available; MapReduce/job committer code ported from HDFS to local.","solutions":["Replace concat with manual concatenation: open each source with fs.open and copy bytes into an FSDataOutputStream created with create(..., overwrite=false) on the destination.","Run the operation against HDFS (DistributedFileSystem implements concat) if the data really lives on the cluster.","Catch UnsupportedOperationException and degrade to the byte-copy path in multi-filesystem code."],"exampleFix":"// before\nfs.concat(target, sources); // throws on LocalFileSystem\n\n// after\ntry (FSDataOutputStream out = fs.create(target, false)) {\n  for (Path src : sources) {\n    try (FSDataInputStream in = fs.open(src)) {\n      IOUtils.copyBytes(in, out, 64 * 1024, false);\n    }\n  }\n}","handlingStrategy":"fallback","validationCode":"if (fs instanceof ChecksumFileSystem) {\n  // concat() unsupported: byte-copy sources into destination instead\n}","typeGuard":"static boolean canConcat(FileSystem fs) {\n  return !(fs instanceof ChecksumFileSystem);\n}","tryCatchPattern":"try {\n  fs.concat(dst, srcs);\n} catch (UnsupportedOperationException e) {\n  try (FSDataOutputStream out = fs.create(dst, true)) {\n    for (Path s : srcs) { try (FSDataInputStream in = fs.open(s)) { IOUtils.copyBytes(in, out, 1<<16, false); } }\n  }\n}","preventionTips":["Treat concat as an HDFS-specific optimization, not a generic FileSystem operation.","Implement concat as open+copy by default and only call fs.concat when you know the implementation supports it."],"tags":["hadoop","filesystem","concat","local-filesystem","checksum-fs","unsupported-operation"],"backgroundTag":"unsupported-filesystem-operation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}