prestodb/presto · error · ParquetCorruptionException

Not valid Parquet file: %s expected magic number: %s or %s,

Error message

Not valid Parquet file: %s expected magic number: %s or %s, but got: %s

What it means

MetadataReader.readFooter() reads the last EXPECTED_FOOTER_SIZE bytes of the file and checks that they end with the Parquet magic number "PAR1" (or "PARE" for encrypted footers). If the trailing 4 bytes match neither, it throws ParquetCorruptionException with a message naming the file id, the expected magic numbers, and the actual bytes found. This is the library's way of saying the object is not a valid (unencrypted or encrypted) Parquet file — the footer is absent, so the file is not Parquet, is truncated, or is a different format.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/cache/MetadataReader.java:124

            throws IOException
    {
        return readFooter(parquetDataSource, fileSize, MODIFICATION_TIME_NOT_SET, fileDecryptor, readMaskedValue);
    }

    public static ParquetFileMetadata readFooter(ParquetDataSource parquetDataSource, long fileSize, long modificationTime, Optional<InternalFileDecryptor> fileDecryptor, boolean readMaskedValue)
            throws IOException
    {
        // Parquet File Layout: https://github.com/apache/parquet-format/blob/master/Encryption.md
        validateParquet(fileSize >= MAGIC.length() + POST_SCRIPT_SIZE, "%s is not a valid Parquet File", parquetDataSource.getId());

        //  EXPECTED_FOOTER_SIZE is an int, so this will never fail
        byte[] buffer = new byte[toIntExact(min(fileSize, EXPECTED_FOOTER_SIZE))];
        parquetDataSource.readFully(fileSize - buffer.length, buffer);
        Slice tailSlice = wrappedBuffer(buffer);

        Slice magic = tailSlice.slice(tailSlice.length() - MAGIC.length(), MAGIC.length());
        if (!MAGIC.equals(magic) && !EMAGIC.equals(magic)) {
            throw new ParquetCorruptionException(format("Not valid Parquet file: %s expected magic number: %s or %s, but got: %s", parquetDataSource.getId(), Arrays.toString(MAGIC.getBytes()), Arrays.toString(EMAGIC.getBytes()), Arrays.toString(magic.getBytes())));
        }
        boolean encryptedFooterMode = EMAGIC.equals(magic);

        int metadataLength = tailSlice.getInt(tailSlice.length() - POST_SCRIPT_SIZE);
        int completeFooterSize = metadataLength + POST_SCRIPT_SIZE;

        long metadataFileOffset = fileSize - completeFooterSize;
        validateParquet(metadataFileOffset >= MAGIC.length() && metadataFileOffset + POST_SCRIPT_SIZE < fileSize, "Corrupted Parquet file: %s metadata index: %s out of range", parquetDataSource.getId(), metadataFileOffset);
        //  Ensure the slice covers the entire metadata range
        if (tailSlice.length() < completeFooterSize) {
            byte[] footerBuffer = new byte[completeFooterSize];
            parquetDataSource.readFully(metadataFileOffset, footerBuffer, 0, footerBuffer.length - tailSlice.length());
            // Copy the previous slice contents into the new buffer
            tailSlice.getBytes(0, footerBuffer, footerBuffer.length - tailSlice.length(), tailSlice.length());
            tailSlice = wrappedBuffer(footerBuffer, 0, footerBuffer.length);
        }

        return readParquetMetadata(tailSlice.slice(tailSlice.length() - completeFooterSize, metadataLength).getInput(), metadataLength, modificationTime, fileDecryptor, encryptedFooterMode, parquetDataSource.getId(), readMaskedValue);

View on GitHub (pinned to 55bb57d202)

Solutions

1. Check the actual file content: `hadoop fs -cat file | tail -c 4` should be PAR1; if not, the file is not valid Parquet.
2. Exclude non-data files from the table location (_SUCCESS, _metadata, dotfiles) or move data files into a clean directory.
3. Re-export/re-copy the file — truncation during transfer (failed S3 multipart, interrupted distcp) is the most common cause.
4. Verify the Hive/connector table's location and format properties point at Parquet data, not another format.
5. Check the file size is greater than the 8-byte minimum (magic + footer length + magic); 0-byte or tiny files always fail this check.

Example fix

// before: blindly scanning table dir
List<String> files = listAll(dir); // includes _SUCCESS, empty files
// -> ParquetCorruptionException: Not valid Parquet file ... got: [83, 69, 71, 10]

// after: filter data files
List<String> files = listAll(dir).stream()
    .filter(f -> !f.startsWith("_") && !f.startsWith(".") && fileSize(f) >= 8)
    .collect(toList());

// Verify the object is Parquet before handing it to the reader
try (FSDataInputStream in = fs.open(path)) {
    long size = fs.getFileStatus(path).getLen();
    if (size < 8) throw new IOException("Too small to be Parquet: " + path);
    byte[] tail = new byte[4];
    in.readFully(size - 4, tail);
    if (!Arrays.equals(tail, "PAR1".getBytes(UTF_8))
            && !Arrays.equals(tail, "PARE".getBytes(UTF_8))) {
        throw new IOException("Not Parquet (bad magic): " + path);
    }
}

try {
    return MetadataReader.readFooter(dataSource, fileSize);
} catch (ParquetCorruptionException e) {
    // bad magic: skip/flag the file instead of failing the whole scan
    logger.warn("Skipping non-Parquet object %s: %s", dataSource.getId(), e.getMessage());
    return skipFile(dataSource);
}

When it happens

Trigger: readFooter() — called recursively or via getParquetMetadata — reads fileSize - footerSize bytes and the last 4 bytes are not PAR1/PARE: the file was not written by a Parquet writer, is truncated (footer bytes lost), has trailing garbage appended, or points at a non-Parquet object (e.g., a directory listing, _SUCCESS marker, or 0-byte file path).

Common situations: Hive table locations containing non-data files (_SUCCESS, .crc, empty files) scanned as Parquet; S3 multipart uploads failing and leaving partial files; gzip/CSV files registered under a Parquet table; copying files with byte truncation; wrong path resolution to an empty object.

Related errors

• We didn't read correct number of definitionLevels
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when
• Still remaining to be read in current batch.
• Corrupted Parquet file: extra %d values to be consumed when

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/505c7c3e20436515. Report an issue: GitHub.