apache/hadoop · error · IOException

compress called on finished compressor

Error message

compress called on finished compressor

What it means

BuiltInGzipCompressor is Hadoop's pure-Java gzip Compressor (used when zlib natives are unavailable and io.compression.codec.gzip.no.native handles the choice). finished() returns true only after the gzip trailer (CRC/size) has been emitted; compress(byte[], off, len) throws IOException("compress called on finished compressor") if invoked after that point, because a gzip member is complete and cannot accept more input. The compressor must be reset() (or reinit(Configuration)) before compressing another stream/member.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java:84

  public BuiltInGzipCompressor(Configuration conf) {
    init(conf);
  }

  @Override
  public boolean finished() {
    // Only if the trailer is also written, it is thought as finished.
    return state == BuiltInGzipDecompressor.GzipStateLabel.FINISHED && deflater.finished();
  }

  @Override
  public boolean needsInput() {
    return deflater.needsInput() && state != BuiltInGzipDecompressor.GzipStateLabel.TRAILER_CRC;
  }

  @Override
  public int compress(byte[] b, int off, int len) throws IOException {
    if (finished()) {
      throw new IOException("compress called on finished compressor");
    }

    if (state == BuiltInGzipDecompressor.GzipStateLabel.ENDED) {
      throw new AlreadyClosedException("compress called on closed compressor");
    }

    int compressedBytesWritten = 0;

    // If we are not within uncompressed data yet, output the header.
    if (state == BuiltInGzipDecompressor.GzipStateLabel.HEADER_BASIC) {
      int outputHeaderSize = writeHeader(b, off, len);
      numExtraBytesWritten += outputHeaderSize;

      compressedBytesWritten += outputHeaderSize;

      if (outputHeaderSize == len) {
        return compressedBytesWritten;
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Before each new input, call compressor.reset() (or reinit(conf) when Configuration-derived settings must be re-applied) — pools typically do this via CodecPool.returnCompressor/CompressorPool reset semantics.
2. Check finished() defensively and reinit when true instead of calling compress on a completed member.
3. Return compressors to CodecPool after each output and check out a fresh/reinit one per file.
4. Audit shared/static compressor fields — make the compressor per-output, not global.

Example fix

// before
compressor.compress(buf, 0, n); // second file: IOException 'compress called on finished compressor'

// after
if (compressor.finished()) {
  compressor.reinit(conf); // or compressor.reset()
}
compressor.compress(buf, 0, n);

if (compressor.finished()) {
  compressor.reinit(conf); // or compressor.reset() when no config is needed
}
compressor.compress(buf, 0, len);

try {
  compressor.compress(buf, 0, len);
} catch (IOException e) {
  if ("compress called on finished compressor".equals(e.getMessage())) {
    compressor.reinit(conf);
    compressor.compress(buf, 0, len); // retry once after reinit
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling compress(...) on a BuiltInGzipCompressor whose state is FINISHED and deflater.finished(): reusing a pooled codec object for a second file without reset(); CompressorStream-like loops that keep calling compress after the trailer was written; multi-file writers sharing one compressor instance.

Common situations: Compressor pools (CodecPool) returning used compressors; frameworks (HDFS sink, Hive writers) that cache codec objects across files/rotations; custom OutputCommitter code writing several gzip outputs through one compressor; tests reusing a static compressor instance.

Related errors

• Compression option provided does not match the file
• All merged files must be compressed or not.
• block overrun
• stream corrupted
• blockSize({}) < 1

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/173c78b571e81227. Report an issue: GitHub.