prestodb/presto · error · OrcCorruptionException

Unexpected end of stream

Error message

Unexpected end of stream

What it means

OrcInputStream.skipFully skips `length` bytes in a loop; if the underlying data source returns a negative result (EOF) before all bytes are skipped, it throws OrcCorruptionException. This means the ORC file's data ended before the metadata/index said it should — the file is truncated or the stream layout is corrupt.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/stream/OrcInputStream.java:171

        if (available() == 0) {
            advance();
            if (buffer == null) {
                return -1;
            }
        }
        length = Math.min(length, available());
        System.arraycopy(buffer, position, b, off, length);
        position += length;
        return length;
    }

    public void skipFully(long length)
            throws IOException
    {
        while (length > 0) {
            long result = skip(length);
            if (result < 0) {
                throw new OrcCorruptionException(orcDataSourceId, "Unexpected end of stream");
            }
            length -= result;
        }
    }

    public void readFully(byte[] buffer, int offset, int length)
            throws IOException
    {
        while (offset < length) {
            int result = read(buffer, offset, length - offset);
            if (result < 0) {
                throw new OrcCorruptionException(orcDataSourceId, "Unexpected end of stream");
            }
            offset += result;
        }
    }

    public OrcDataSourceId getOrcDataSourceId()

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify the file size and completeness (compare checksums, ensure writer finished/committed).
2. Re-read the file with fresh metadata (clear caches) in case offsets came from a stale/other file.
3. Check for truncated downloads/uploads; re-transfer the file.
4. Check that compression/chunk boundaries are not being skipped with wrong offsets (reader/writer version mismatch).

Example fix

// before: reading truncated file copied from HDFS
// after: verify integrity before opening
long srcLen = orcDataSource.getEstimatedSize();
if (srcLen < expectedFooterEnd) {
    throw new IOException("ORC file truncated: " + orcDataSourceId);
}

long expected = footerMetadataEndOffset;
long actual = orcDataSource.getLength();
if (actual < expected) throw new IOException("ORC truncated: " + orcDataSourceId);

try { stream.skipFully(len); } catch (OrcCorruptionException e) {
    log.error("ORC stream truncated for %s", e.getOrcDataSourceId(), e);
    throw new DataReadException("source file incomplete", e);
}

When it happens

Trigger: skipFully called while repositioning a value stream past data; underlying OrcDataSource has no more bytes but more skipping remains; typically a truncated file or wrong stripe/chunk offset.

Common situations: Incomplete ORC file upload or copy (S3/HDFS partial read); reader metadata points beyond actual data; wrong file used with cached stripe/footer metadata; network read cut short.

Related errors

• Read past end of RLE integer
• End of stream in RLE Integer
• Validation failed
• Read past end of buffer RLE byte
• Reading RLE byte got EOF

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/530f7103b3e180f7. Report an issue: GitHub.