prestodb/presto · error · OrcCorruptionException

Invalid file size %s

Error message

Invalid file size %s

What it means

This library throws OrcCorruptionException when an ORC data source is too small to be a valid ORC file. A file must at least contain the 3-byte magic header plus a postscript byte, so a file of size <= MAGIC.length() (3) cannot hold the ORC file tail. StorageOrcFileTailSource.getOrcFileTail checks this before reading the tail bytes.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/cache/StorageOrcFileTailSource.java:72

    public StorageOrcFileTailSource()
    {
        this(EXPECTED_FOOTER_SIZE_IN_BYTES, false);
    }

    public StorageOrcFileTailSource(int expectedFooterSizeInBytes, boolean dwrfStripeCacheEnabled)
    {
        checkArgument(expectedFooterSizeInBytes >= MINIMUM_TAIL_SIZE_IN_BYTES, "expectedFooterSize %s is less than minimum supported", expectedFooterSizeInBytes);
        this.expectedFooterSizeInBytes = expectedFooterSizeInBytes;
        this.dwrfStripeCacheEnabled = dwrfStripeCacheEnabled;
    }

    @Override
    public OrcFileTail getOrcFileTail(OrcDataSource orcDataSource, MetadataReader metadataReader, Optional<OrcWriteValidation> writeValidation, boolean cacheable, long fileModificationTime)
            throws IOException
    {
        long size = orcDataSource.getSize();
        if (size <= MAGIC.length()) {
            throw new OrcCorruptionException(orcDataSource.getId(), "Invalid file size %s", size);
        }

        // Read the tail of the file
        byte[] buffer = new byte[toIntExact(min(size, expectedFooterSizeInBytes))];
        orcDataSource.readFully(size - buffer.length, buffer);

        // get length of PostScript - last byte of the file
        int postScriptSize = buffer[buffer.length - SIZE_OF_BYTE] & 0xff;
        if (postScriptSize >= buffer.length) {
            throw new OrcCorruptionException(orcDataSource.getId(), "Invalid postscript length %s", postScriptSize);
        }

        // decode the post script
        PostScript postScript;
        try {
            postScript = metadataReader.readPostScript(buffer, buffer.length - SIZE_OF_BYTE - postScriptSize, postScriptSize);
        }
        catch (OrcCorruptionException e) {

View on GitHub (pinned to 55bb57d202)

Solutions

  1. Verify the file exists and has a non-trivial size (ls -l / hdfs dfs -stat) before reading; re-copy or re-download if truncated.
  2. Confirm the file is a real ORC file (starts with 'ORC' and has a postscript); re-run the writer job if the file is incomplete.
  3. Wait for the writer to finish/commit before querying files from the output directory.
  4. If the file is intentionally empty, skip it rather than reading it as ORC.

Example fix

// before
OrcFileTail tail = fileTailSource.getOrcFileTail(orcDataSource, metadataReader, Optional.empty(), false, 0);
// after
if (orcDataSource.getSize() <= 3) {
    log.warn("Skipping %s: not a valid ORC file (size=%s)", orcDataSource.getId(), orcDataSource.getSize());
    return;
}
OrcFileTail tail = fileTailSource.getOrcFileTail(orcDataSource, metadataReader, Optional.empty(), false, 0);
Defensive patterns

Strategy: validation

Validate before calling

if (orcDataSource.getSize() <= OrcFile.Magic.length()) {
    throw new SkipFileException("File too small to be ORC: " + orcDataSource.getId());
}

Try / catch

try { tail = source.getOrcFileTail(ds, reader, Optional.empty(), false, time); }
catch (OrcCorruptionException e) { log.warn("Skipping unreadable ORC file %s", ds.getId(), e); }

Prevention

When it happens

Trigger: Calling getOrcFileTail (directly or via orcFileTail) on an OrcDataSource whose getSize() is 0, 1, 2, or 3 bytes, e.g. a truncated, empty, or placeholder file passed to the ORC tail-source.

Common situations: Reading a file still being written (footer not flushed), a truncated download/copy, an empty file created by a failed job, or pointing at a non-ORC placeholder file (e.g. _SUCCESS, .crc).

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/f099c3d5f5f7bd63. Report an issue: GitHub.