{"record":{"id":"505c7c3e20436515","repo":"prestodb/presto","slug":"not-valid-parquet-file-s-expected-magic-number","errorCode":null,"errorMessage":"Not valid Parquet file: %s expected magic number: %s or %s, but got: %s","messagePattern":"Not valid Parquet file: (.+?) expected magic number: (.+?) or (.+?), but got: (.+?)","errorType":"exception","errorClass":"ParquetCorruptionException","httpStatus":null,"severity":"error","filePath":"presto-parquet/src/main/java/com/facebook/presto/parquet/cache/MetadataReader.java","lineNumber":124,"sourceCode":"            throws IOException\n    {\n        return readFooter(parquetDataSource, fileSize, MODIFICATION_TIME_NOT_SET, fileDecryptor, readMaskedValue);\n    }\n\n    public static ParquetFileMetadata readFooter(ParquetDataSource parquetDataSource, long fileSize, long modificationTime, Optional<InternalFileDecryptor> fileDecryptor, boolean readMaskedValue)\n            throws IOException\n    {\n        // Parquet File Layout: https://github.com/apache/parquet-format/blob/master/Encryption.md\n        validateParquet(fileSize >= MAGIC.length() + POST_SCRIPT_SIZE, \"%s is not a valid Parquet File\", parquetDataSource.getId());\n\n        //  EXPECTED_FOOTER_SIZE is an int, so this will never fail\n        byte[] buffer = new byte[toIntExact(min(fileSize, EXPECTED_FOOTER_SIZE))];\n        parquetDataSource.readFully(fileSize - buffer.length, buffer);\n        Slice tailSlice = wrappedBuffer(buffer);\n\n        Slice magic = tailSlice.slice(tailSlice.length() - MAGIC.length(), MAGIC.length());\n        if (!MAGIC.equals(magic) && !EMAGIC.equals(magic)) {\n            throw new ParquetCorruptionException(format(\"Not valid Parquet file: %s expected magic number: %s or %s, but got: %s\", parquetDataSource.getId(), Arrays.toString(MAGIC.getBytes()), Arrays.toString(EMAGIC.getBytes()), Arrays.toString(magic.getBytes())));\n        }\n        boolean encryptedFooterMode = EMAGIC.equals(magic);\n\n        int metadataLength = tailSlice.getInt(tailSlice.length() - POST_SCRIPT_SIZE);\n        int completeFooterSize = metadataLength + POST_SCRIPT_SIZE;\n\n        long metadataFileOffset = fileSize - completeFooterSize;\n        validateParquet(metadataFileOffset >= MAGIC.length() && metadataFileOffset + POST_SCRIPT_SIZE < fileSize, \"Corrupted Parquet file: %s metadata index: %s out of range\", parquetDataSource.getId(), metadataFileOffset);\n        //  Ensure the slice covers the entire metadata range\n        if (tailSlice.length() < completeFooterSize) {\n            byte[] footerBuffer = new byte[completeFooterSize];\n            parquetDataSource.readFully(metadataFileOffset, footerBuffer, 0, footerBuffer.length - tailSlice.length());\n            // Copy the previous slice contents into the new buffer\n            tailSlice.getBytes(0, footerBuffer, footerBuffer.length - tailSlice.length(), tailSlice.length());\n            tailSlice = wrappedBuffer(footerBuffer, 0, footerBuffer.length);\n        }\n\n        return readParquetMetadata(tailSlice.slice(tailSlice.length() - completeFooterSize, metadataLength).getInput(), metadataLength, modificationTime, fileDecryptor, encryptedFooterMode, parquetDataSource.getId(), readMaskedValue);","sourceCodeStart":106,"sourceCodeEnd":142,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-parquet/src/main/java/com/facebook/presto/parquet/cache/MetadataReader.java#L106-L142","documentation":"MetadataReader.readFooter() reads the last EXPECTED_FOOTER_SIZE bytes of the file and checks that they end with the Parquet magic number \"PAR1\" (or \"PARE\" for encrypted footers). If the trailing 4 bytes match neither, it throws ParquetCorruptionException with a message naming the file id, the expected magic numbers, and the actual bytes found. This is the library's way of saying the object is not a valid (unencrypted or encrypted) Parquet file — the footer is absent, so the file is not Parquet, is truncated, or is a different format.","triggerScenarios":"readFooter() — called recursively or via getParquetMetadata — reads fileSize - footerSize bytes and the last 4 bytes are not PAR1/PARE: the file was not written by a Parquet writer, is truncated (footer bytes lost), has trailing garbage appended, or points at a non-Parquet object (e.g., a directory listing, _SUCCESS marker, or 0-byte file path).","commonSituations":"Hive table locations containing non-data files (_SUCCESS, .crc, empty files) scanned as Parquet; S3 multipart uploads failing and leaving partial files; gzip/CSV files registered under a Parquet table; copying files with byte truncation; wrong path resolution to an empty object.","solutions":["Check the actual file content: `hadoop fs -cat file | tail -c 4` should be PAR1; if not, the file is not valid Parquet.","Exclude non-data files from the table location (_SUCCESS, _metadata, dotfiles) or move data files into a clean directory.","Re-export/re-copy the file — truncation during transfer (failed S3 multipart, interrupted distcp) is the most common cause.","Verify the Hive/connector table's location and format properties point at Parquet data, not another format.","Check the file size is greater than the 8-byte minimum (magic + footer length + magic); 0-byte or tiny files always fail this check."],"exampleFix":"// before: blindly scanning table dir\nList<String> files = listAll(dir); // includes _SUCCESS, empty files\n// -> ParquetCorruptionException: Not valid Parquet file ... got: [83, 69, 71, 10]\n\n// after: filter data files\nList<String> files = listAll(dir).stream()\n    .filter(f -> !f.startsWith(\"_\") && !f.startsWith(\".\") && fileSize(f) >= 8)\n    .collect(toList());","handlingStrategy":"validation","validationCode":"// Verify the object is Parquet before handing it to the reader\ntry (FSDataInputStream in = fs.open(path)) {\n    long size = fs.getFileStatus(path).getLen();\n    if (size < 8) throw new IOException(\"Too small to be Parquet: \" + path);\n    byte[] tail = new byte[4];\n    in.readFully(size - 4, tail);\n    if (!Arrays.equals(tail, \"PAR1\".getBytes(UTF_8))\n            && !Arrays.equals(tail, \"PARE\".getBytes(UTF_8))) {\n        throw new IOException(\"Not Parquet (bad magic): \" + path);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    return MetadataReader.readFooter(dataSource, fileSize);\n} catch (ParquetCorruptionException e) {\n    // bad magic: skip/flag the file instead of failing the whole scan\n    logger.warn(\"Skipping non-Parquet object %s: %s\", dataSource.getId(), e.getMessage());\n    return skipFile(dataSource);\n}","preventionTips":["Filter out non-data files (_SUCCESS, .crc, dotfiles) from table locations before scanning.","Check trailing 4 bytes == PAR1 when ingesting external files into a Parquet table.","Ensure transfers (S3 multipart, distcp) complete and verify sizes/etags.","Never point a Parquet table at directories containing empty or foreign-format files."],"tags":["parquet","corrupt-file","magic-number","file-format"],"backgroundTag":"invalid-parquet-magic-number","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}