What causes "Compressed stream is truncated" in prestodb/presto?

Validate/re-copy the source file and compare checksums against the original.

prestodb/presto · error · RcFileCorruptionException

Compressed stream is truncated

Error message

Compressed stream is truncated

What it means

AircompressorDecompressor.decompress uses the aircompressor codec to inflate an RCFile compressed block. If the decompression stream ends prematurely (IOException or IndexOutOfBoundsException), it throws RcFileCorruptionException 'Compressed stream is truncated'. The on-disk block does not contain the full compressed payload it claims.

Source

Thrown at presto-rcfile/src/main/java/com/facebook/presto/rcfile/AircompressorDecompressor.java:42

public class AircompressorDecompressor
        implements RcFileDecompressor
{
    private final CompressionCodec codec;

    public AircompressorDecompressor(CompressionCodec codec)
    {
        this.codec = requireNonNull(codec, "codec is null");
    }

    @Override
    public void decompress(Slice compressed, Slice uncompressed)
            throws RcFileCorruptionException
    {
        try (CompressionInputStream decompressorStream = codec.createInputStream(compressed.getInput())) {
            uncompressed.setBytes(0, decompressorStream, uncompressed.length());
        }
        catch (IndexOutOfBoundsException | IOException e) {
            throw new RcFileCorruptionException(e, "Compressed stream is truncated");
        }
    }

    @Override
    public void destroy()
    {
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

Validate/re-copy the source file and compare checksums against the original.
Identify which job produced the file and re-run it; confirm writes completed.
Check reader/writer version compatibility for the compression codec.
If only some stripes are bad, skip corrupt stripes if your reader supports it.

Defensive patterns

Strategy: try-catch

Validate before calling

// validate file completeness before reading (size/sync-marker sanity)
FileSystem fs = path.getFileSystem(conf);
long len = fs.getFileStatus(path).getLen();
if (len == 0 || len < expectedMinimumBlockSize) throw new IllegalStateException("File suspiciously small/truncated: " + path);

Try / catch

try {
  rcFileReader.readBlock(block);
} catch (RcFileCorruptionException e) {
  if (String.valueOf(e.getMessage()).contains("Compressed stream is truncated")) {
    log.error("Corrupt RCFile block, skipping or re-fetching source: " + e.getCause());
  } else throw e;
}

Prevention

Verify checksums (HDFS does this; check local FS too) before reading files
Never copy files before the writing job completes and closes them
Re-run producing jobs on failure instead of consuming partial output
Keep aircompressor/Hadoop versions consistent between writers and readers

When it happens

Trigger: Reading an RCFile whose compressed block bytes are cut short — the codec's CompressionInputStream hits EOF before producing the expected uncompressed.length() bytes.

Common situations: Files corrupted by failed/truncated writes (job killed mid-write, partial HDFS block); wrong byte offsets/sync markers from a writer bug; copying files while they were still being written.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/43eef12b4e73724f. Report an issue: GitHub.