apache/hadoop · error · IOException

All merged files must be compressed or not.

Error message

All merged files must be compressed or not.

What it means

SequenceFile.Sorter's MergeQueue fixes the expected compression mode from the first segment added, then requires every subsequent segment to match both the compress flag and the blockCompress flag. Any later file that differs (compressed vs uncompressed, or RECORD vs BLOCK compression) throws IOException("All merged files must be compressed or not."). Mixing compression modes would make a single valid output header impossible.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:3517

      private float progPerByte;
      private Progress mergeProgress = new Progress();
      private Path tmpDir;
      private Progressable progress = null; //handle to the progress reporting object
      private SegmentDescriptor minSegment;
      
      //a TreeMap used to store the segments sorted by size (segment offset and
      //segment path name is used to break ties between segments of same sizes)
      private Map<SegmentDescriptor, Void> sortedSegmentSizes =
        new TreeMap<SegmentDescriptor, Void>();
            
      @SuppressWarnings("unchecked")
      public void put(SegmentDescriptor stream) throws IOException {
        if (size() == 0) {
          compress = stream.in.isCompressed();
          blockCompress = stream.in.isBlockCompressed();
        } else if (compress != stream.in.isCompressed() || 
                   blockCompress != stream.in.isBlockCompressed()) {
          throw new IOException("All merged files must be compressed or not.");
        } 
        super.put(stream);
      }
      
      /**
       * A queue of file segments to merge
       * @param segments the file segments to merge
       * @param tmpDir a relative local directory to save intermediate files in
       * @param progress the reference to the Progressable object
       */
      public MergeQueue(List <SegmentDescriptor> segments,
          Path tmpDir, Progressable progress) {
        int size = segments.size();
        for (int i = 0; i < size; i++) {
          sortedSegmentSizes.put(segments.get(i), null);
        }
        this.tmpDir = tmpDir;
        this.progress = progress;

View on GitHub (pinned to 2add963021)

Solutions

1. Partition the input list by compression mode (read each header with a short-lived SequenceFile.Reader; use isCompressed()/isBlockCompressed()) and run one merge per group.
2. Or normalize first: rewrite every input with one chosen CompressionType and codec, then merge.
3. Set identical compression configuration on all producing jobs so inputs can never diverge.

Example fix

// before
sorter.merge(inFiles, outFile);           // throws when modes differ

// after: group by compression mode, merge each group separately
Map<List<Boolean>, List<Path>> groups = new HashMap<>();
for (Path p : inFiles) {
  try (SequenceFile.Reader r = new SequenceFile.Reader(conf,
      SequenceFile.Reader.file(p))) {
    groups.computeIfAbsent(
        Arrays.asList(r.isCompressed(), r.isBlockCompressed()),
        k -> new ArrayList<>()).add(p);
  }
}
for (List<Path> group : groups.values()) {
  sorter.merge(group.toArray(new Path[0]), nextUniqueOutput());
}

static boolean homogeneousCompression(Configuration conf, Path[] inFiles)
    throws IOException {
  Boolean compress = null, block = null;
  for (Path p : inFiles) {
    try (SequenceFile.Reader r = new SequenceFile.Reader(conf,
        SequenceFile.Reader.file(p))) {
      if (compress == null) {
        compress = r.isCompressed();
        block = r.isBlockCompressed();
      } else if (compress != r.isCompressed()
                 || block != r.isBlockCompressed()) {
        return false;
      }
    }
  }
  return true;
}

When it happens

Trigger: Calling Sorter.sort/merge/sortAndIterate over inputs where some SequenceFiles were written with CompressionType.RECORD or BLOCK and others with NONE; mixing RECORD- and BLOCK-compressed files (blockCompress flag differs even if both are 'compressed'); merging map outputs from jobs with different mapreduce.map.output.compression settings.

Common situations: Merging historical outputs written under older configs; inputs produced by different tools (some compress, some don't); changing io.seqfile.compression.type between runs and then sorting the combined set.

Related errors

• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• WritableValueBytes doesn't support RECORD compression
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• Compression option provided does not match the file
• Unsupported call for block-compressed SequenceFiles - use Se

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/de673d2c2f980308. Report an issue: GitHub.