{"record":{"id":"b083f5803243f652","repo":"apache/hadoop","slug":"concat-source-file-src-has-preferred-blo","errorCode":null,"errorMessage":"\"concat: source file \" + src + \" has preferred block size \" + srcINodeFile.getPreferredBlockSize() + \" which is greater than the target file's preferred block size \" + targetINode.getPreferredBlockSize()","messagePattern":"\"concat: source file \" \\+ src \\+ \" has preferred block size \" \\+ srcINodeFile\\.getPreferredBlockSize\\(\\) \\+ \" which is greater than the target file's preferred block size \" \\+ targetINode\\.getPreferredBlockSize\\(\\)","errorType":"exception","errorClass":"HadoopIllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java","lineNumber":162,"sourceCode":"        throw new SnapshotException(\"Concat: the source file \" + src\n            + \" is referred by some other reference in some snapshot.\");\n      }\n      // source file cannot be the same with the target file\n      if (srcINode.equals(targetINode)) {\n        throw new HadoopIllegalArgumentException(\"concat: the src file \" + src\n            + \" is the same with the target file \" + targetIIP.getPath());\n      }\n      // source file cannot be under construction or empty\n      if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" is invalid or empty or underConstruction\");\n      }\n\n      // source file's preferred block size cannot be greater than the target\n      // file\n      if (srcINodeFile.getPreferredBlockSize() >\n          targetINode.getPreferredBlockSize()) {\n        throw new HadoopIllegalArgumentException(\"concat: source file \" + src\n            + \" has preferred block size \" + srcINodeFile.getPreferredBlockSize()\n            + \" which is greater than the target file's preferred block size \"\n            + targetINode.getPreferredBlockSize());\n      }\n      if(srcINodeFile.getErasureCodingPolicyID() !=\n          targetINode.getErasureCodingPolicyID()) {\n        throw new HadoopIllegalArgumentException(\"Source file \" + src\n            + \" and target file \" + targetIIP.getPath()\n            + \" have different erasure coding policy\");\n      }\n      si.add(srcINodeFile);\n    }\n\n    // make sure no two files are the same\n    if(si.size() < srcs.length) {\n      // it means at least two files are the same\n      throw new HadoopIllegalArgumentException(\n          \"concat: at least two of the source files are the same\");","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java#L144-L180","documentation":"A source file's preferred block size may not exceed the target's: verifySrcFiles compares getPreferredBlockSize() and throws HadoopIllegalArgumentException when src > target. Concat simply appends the src's blocks to the target, so a larger source block would sit inside a file declared with a smaller block size, breaking readers that compute block offsets from the target's block size. (Src equal to target is allowed.)","triggerScenarios":"Concat where src files were created with dfs.block.size (or per-create blockSize) larger than the target's — e.g. target created with default 128m but sources written with 256m, or vice versa after a config change.","commonSituations":"Cluster-wide block-size changes between when the target and the sources were written; jobs that create files with explicit blockSize parameters while the compaction target uses the client default; merging files produced by different pipelines.","solutions":["Choose the file with the largest block size as the concat target, or recreate the target with blockSize >= max(src block sizes).","Recreate the oversized src files with the target's block size (rewrite via copy) before concat.","Standardize dfs.block.size across the pipeline creating these files so target and srcs always match."],"exampleFix":"// before\nfs.concat(new Path(\"/data/part-0000\"), srcs); // part-0000 has 128m blocks, a src has 256m\n\n// after\n// pick the largest-block file as target\nPath target = dirFiles.stream()\n    .max(Comparator.comparingLong(p -> fs.getFileStatus(p).getBlockSize()))\n    .get();\nfs.concat(target, srcsExcluding(target, dirFiles));","handlingStrategy":"validation","validationCode":"long targetBs = fs.getFileStatus(target).getBlockSize();\nfor (Path src : srcs) {\n  if (fs.getFileStatus(src).getBlockSize() > targetBs) {\n    throw new IllegalStateException(src + \" block size exceeds target's \" + targetBs);\n  }\n}","typeGuard":null,"tryCatchPattern":"catch (HadoopIllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"preferred block size\")) {\n    // choose the largest-block file as target and re-run\n    fs.concat(largestBlockSizeFile(dir), srcsExcludingIt(dir));\n  } else { throw e; }\n}","preventionTips":["Standardize dfs.block.size (and explicit create() blockSize) across all writers feeding a concat target.","During compaction planning, compare getBlockSize() of candidates and pick the largest as target.","After any cluster block-size change, expect old files to violate this and plan rewrites."],"tags":["hdfs","concat","block-size","precondition","configuration-mismatch"],"backgroundTag":"concat-block-size-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}