apache/hadoop · error · UnsupportedOperationException

Concat is not supported by ChecksumFileSystem

Error message

Concat is not supported by ChecksumFileSystem

What it means

ChecksumFileSystem.concat(Path, Path[]) unconditionally throws UnsupportedOperationException. Concat (server-side concatenation of files without copying) only exists on filesystems with block-level semantics like HDFS; a checksumming local wrapper cannot splice the per-chunk .crc files of the sources, so it refuses the operation.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java:654

    return new FSDataBoundedInputStream(fs, f, in);
  }

  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException("Append is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public boolean truncate(Path f, long newLength) throws IOException {
    throw new UnsupportedOperationException("Truncate is not supported "
        + "by ChecksumFileSystem");
  }

  @Override
  public void concat(final Path f, final Path[] psrcs) throws IOException {
    throw new UnsupportedOperationException("Concat is not supported "
        + "by ChecksumFileSystem");
  }

  /**
   * Calculated the length of the checksum file in bytes.
   * @param size the length of the data file in bytes
   * @param bytesPerSum the number of bytes in a checksum block
   * @return the number of bytes in the checksum file
   */
  public static long getChecksumLength(long size, int bytesPerSum) {
    //the checksum length is equal to size passed divided by bytesPerSum +
    //bytes written in the beginning of the checksum file.
    return ((size + bytesPerSum - 1) / bytesPerSum) * FSInputChecker.CHECKSUM_SIZE +
             ChecksumFSInputChecker.HEADER_LENGTH;
  }

  /** This class provides an output stream for a checksummed file.
   * It generates checksums for data. */

View on GitHub (pinned to 2add963021)

Solutions

  1. Replace concat with manual concatenation: open each source with fs.open and copy bytes into an FSDataOutputStream created with create(..., overwrite=false) on the destination.
  2. Run the operation against HDFS (DistributedFileSystem implements concat) if the data really lives on the cluster.
  3. Catch UnsupportedOperationException and degrade to the byte-copy path in multi-filesystem code.

Example fix

// before
fs.concat(target, sources); // throws on LocalFileSystem

// after
try (FSDataOutputStream out = fs.create(target, false)) {
  for (Path src : sources) {
    try (FSDataInputStream in = fs.open(src)) {
      IOUtils.copyBytes(in, out, 64 * 1024, false);
    }
  }
}
Defensive patterns

Strategy: fallback

Validate before calling

if (fs instanceof ChecksumFileSystem) {
  // concat() unsupported: byte-copy sources into destination instead
}

Type guard

static boolean canConcat(FileSystem fs) {
  return !(fs instanceof ChecksumFileSystem);
}

Try / catch

try {
  fs.concat(dst, srcs);
} catch (UnsupportedOperationException e) {
  try (FSDataOutputStream out = fs.create(dst, true)) {
    for (Path s : srcs) { try (FSDataInputStream in = fs.open(s)) { IOUtils.copyBytes(in, out, 1<<16, false); } }
  }
}

Prevention

When it happens

Trigger: Calling concat(dst, srcs) on LocalFileSystem or any ChecksumFileSystem subclass, typically in code written against DistributedFileSystem.concat and pointed at file:// paths (unit tests, local shuffle merging, distcp between HDFS and file://).

Common situations: Test code reusing HDFS concat logic against local mini-clusters/fixtures; utilities that treat FileSystem.concat as universally available; MapReduce/job committer code ported from HDFS to local.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/73f52c8ea695fbc9. Report an issue: GitHub.