apache/hadoop · error · HadoopIllegalArgumentException

"concat: source file " + src + " has preferred block size "

Error message

"concat: source file " + src + " has preferred block size " + srcINodeFile.getPreferredBlockSize() + " which is greater than the target file's preferred block size " + targetINode.getPreferredBlockSize()

What it means

A source file's preferred block size may not exceed the target's: verifySrcFiles compares getPreferredBlockSize() and throws HadoopIllegalArgumentException when src > target. Concat simply appends the src's blocks to the target, so a larger source block would sit inside a file declared with a smaller block size, breaking readers that compute block offsets from the target's block size. (Src equal to target is allowed.)

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java:162

        throw new SnapshotException("Concat: the source file " + src
            + " is referred by some other reference in some snapshot.");
      }
      // source file cannot be the same with the target file
      if (srcINode.equals(targetINode)) {
        throw new HadoopIllegalArgumentException("concat: the src file " + src
            + " is the same with the target file " + targetIIP.getPath());
      }
      // source file cannot be under construction or empty
      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " is invalid or empty or underConstruction");
      }

      // source file's preferred block size cannot be greater than the target
      // file
      if (srcINodeFile.getPreferredBlockSize() >
          targetINode.getPreferredBlockSize()) {
        throw new HadoopIllegalArgumentException("concat: source file " + src
            + " has preferred block size " + srcINodeFile.getPreferredBlockSize()
            + " which is greater than the target file's preferred block size "
            + targetINode.getPreferredBlockSize());
      }
      if(srcINodeFile.getErasureCodingPolicyID() !=
          targetINode.getErasureCodingPolicyID()) {
        throw new HadoopIllegalArgumentException("Source file " + src
            + " and target file " + targetIIP.getPath()
            + " have different erasure coding policy");
      }
      si.add(srcINodeFile);
    }

    // make sure no two files are the same
    if(si.size() < srcs.length) {
      // it means at least two files are the same
      throw new HadoopIllegalArgumentException(
          "concat: at least two of the source files are the same");

View on GitHub (pinned to 2add963021)

Solutions

Choose the file with the largest block size as the concat target, or recreate the target with blockSize >= max(src block sizes).
Recreate the oversized src files with the target's block size (rewrite via copy) before concat.
Standardize dfs.block.size across the pipeline creating these files so target and srcs always match.

Example fix

// before
fs.concat(new Path("/data/part-0000"), srcs); // part-0000 has 128m blocks, a src has 256m

// after
// pick the largest-block file as target
Path target = dirFiles.stream()
    .max(Comparator.comparingLong(p -> fs.getFileStatus(p).getBlockSize()))
    .get();
fs.concat(target, srcsExcluding(target, dirFiles));

Defensive patterns

Strategy: validation

Validate before calling

long targetBs = fs.getFileStatus(target).getBlockSize();
for (Path src : srcs) {
  if (fs.getFileStatus(src).getBlockSize() > targetBs) {
    throw new IllegalStateException(src + " block size exceeds target's " + targetBs);
  }
}

Try / catch

catch (HadoopIllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("preferred block size")) {
    // choose the largest-block file as target and re-run
    fs.concat(largestBlockSizeFile(dir), srcsExcludingIt(dir));
  } else { throw e; }
}

Prevention

Standardize dfs.block.size (and explicit create() blockSize) across all writers feeding a concat target.
During compaction planning, compare getBlockSize() of candidates and pick the largest as target.
After any cluster block-size change, expect old files to violate this and plan rewrites.

When it happens

Trigger: Concat where src files were created with dfs.block.size (or per-create blockSize) larger than the target's — e.g. target created with default 128m but sources written with 256m, or vice versa after a config change.

Common situations: Cluster-wide block-size changes between when the target and the sources were written; jobs that create files with explicit blockSize parameters while the compaction target uses the client default; merging files produced by different pipelines.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b083f5803243f652. Report an issue: GitHub.