apache/hadoop · error · IOException

Unsupported call for block-compressed SequenceFiles - use Se

Error message

Unsupported call for block-compressed SequenceFiles - use SequenceFile.Reader.next(DataOutputStream, ValueBytes)

What it means

The deprecated SequenceFile.Reader.next(DataOutputBuffer) API predates block compression: BLOCK-compressed SequenceFiles pack many keys and values into per-block streams (keyLenIn/keyIn/valLenIn/valIn), so a method that appends one raw record into a single buffer cannot serve them. It checks the blockCompressed flag up front and throws IOException telling you to use next(DataOutputStream, ValueBytes), i.e. the raw API nextRaw(DataOutputBuffer, ValueBytes) or the typed next(key, value).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:2569

        }
        length = in.readInt();                  // re-read length
      } else {
        syncSeen = false;
      }
      
      return length;
    }
    
    /** Read the next key/value pair in the file into <code>buffer</code>.
     * Returns the length of the key read, or -1 if at end of file.  The length
     * of the value may be computed by calling buffer.getLength() before and
     * after calls to this method. */
    /** @deprecated Call {@link #nextRaw(DataOutputBuffer,SequenceFile.ValueBytes)}. */
    @Deprecated
    synchronized int next(DataOutputBuffer buffer) throws IOException {
      // Unsupported for block-compressed sequence files
      if (blockCompressed) {
        throw new IOException("Unsupported call for block-compressed" +
                              " SequenceFiles - use SequenceFile.Reader.next(DataOutputStream, ValueBytes)");
      }
      try {
        int length = readRecordLength();
        if (length == -1) {
          return -1;
        }
        int keyLength = in.readInt();
        buffer.write(in, length);
        return keyLength;
      } catch (ChecksumException e) {             // checksum failure
        handleChecksumException(e);
        return next(buffer);
      }
    }

    public ValueBytes createValueBytes() {
      ValueBytes val = null;

View on GitHub (pinned to 2add963021)

Solutions

1. Replace next(DataOutputBuffer) with nextRaw(DataOutputBuffer, ValueBytes) — obtain the ValueBytes from reader.createValueBytes() — or use the typed next(WritableComparable, Writable), both of which handle all compression types.
2. If the raw-buffer code path cannot be changed, rewrite the input file with CompressionType.RECORD or NONE instead of BLOCK.
3. Audit for other deprecated calls in the same reader code path; block compression also breaks other record-oriented assumptions.

Example fix

// before
DataOutputBuffer buf = new DataOutputBuffer();
int keyLen = reader.next(buf);          // throws on BLOCK-compressed files

// after
DataOutputBuffer keyBuf = new DataOutputBuffer();
SequenceFile.ValueBytes val = reader.createValueBytes();
int recLen = reader.nextRaw(keyBuf, val);  // works for NONE/RECORD/BLOCK

if (reader.isBlockCompressed()) {
  // BLOCK-compressed: use the raw or typed API that supports blocks
  SequenceFile.ValueBytes val = reader.createValueBytes();
  int recLen = reader.nextRaw(keyBuf, val);
} else {
  int keyLen = reader.next(buffer);  // deprecated, record-based files only
}

static boolean supportsRawBufferRead(SequenceFile.Reader r) {
  return !r.isBlockCompressed();
}

When it happens

Trigger: Calling the package-visible deprecated next(DataOutputBuffer) (directly, or via next(Object key) internals on non-block files) on a file written with CompressionType.BLOCK. Typical producer settings: io.seqfile.compression.type=BLOCK or mapreduce.map.output.compression.type=BLOCK, then consuming it with legacy raw-buffer RecordReader code.

Common situations: Legacy custom InputFormats or third-party libraries compiled against old Hadoop APIs; jobs whose intermediate/map-output compression was switched to BLOCK between versions; old MapFile utilities that reused the raw buffer API.

Related errors

• zero length key found!
• Missing parent:${f}
• already exists: {}
• All merged files must be compressed or not.
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cffef6fe75bebaa5. Report an issue: GitHub.