{"record":{"id":"88fd91a9007b1e35","repo":"prestodb/presto","slug":"malformed-hive-file-statistics-88fd91","errorCode":"MALFORMED_HIVE_FILE_STATISTICS","errorMessage":"Filename = %s not stored in manifest. Partition = %s, TableName = %s","messagePattern":"Filename = (.+?) not stored in manifest\\. Partition = (.+?), TableName = (.+?)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/ManifestPartitionLoader.java","lineNumber":208,"sourceCode":"    private void validateManifest(ConnectorSession session, HivePartitionMetadata partition, Path path, List<String> manifestFileNames, List<Long> manifestFileSizes)\n            throws IOException\n    {\n        ExtendedFileSystem fileSystem = hdfsEnvironment.getFileSystem(hdfsContext, path);\n        HiveDirectoryContext hiveDirectoryContext = new HiveDirectoryContext(\n                recursiveDirWalkerEnabled ? RECURSE : IGNORED,\n                false,\n                isSkipEmptyFilesEnabled(session),\n                hdfsContext.getIdentity(),\n                buildDirectoryContextProperties(session),\n                session.getRuntimeStats());\n\n        Iterator<HiveFileInfo> fileInfoIterator = directoryLister.list(fileSystem, table, path, partition.getPartition(), namenodeStats, hiveDirectoryContext);\n        int fileCount = 0;\n        while (fileInfoIterator.hasNext()) {\n            HiveFileInfo fileInfo = fileInfoIterator.next();\n            String fileName = fileInfo.getFileName();\n            if (!manifestFileNames.contains(fileName)) {\n                throw new PrestoException(\n                        MALFORMED_HIVE_FILE_STATISTICS,\n                        format(\"Filename = %s not stored in manifest. Partition = %s, TableName = %s\",\n                                fileName,\n                                partition.getHivePartition().getPartitionId(),\n                                table.getTableName()));\n            }\n\n            int index = manifestFileNames.indexOf(fileName);\n            if (!manifestFileSizes.get(index).equals(fileInfo.getLength())) {\n                throw new PrestoException(\n                        MALFORMED_HIVE_FILE_STATISTICS,\n                        format(\"FilesizeFromManifest = %s is not equal to FilesizeFromStorage = %s. File = %s, Partition = %s, TableName = %s\",\n                                manifestFileSizes.get(index),\n                                fileInfo.getLength(),\n                                fileName,\n                                partition.getHivePartition().getPartitionId(),\n                                table.getTableName()));\n            }","sourceCodeStart":190,"sourceCodeEnd":226,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/ManifestPartitionLoader.java#L190-L226","documentation":"When a partition is loaded with manifest-based file statistics, Presto validates that every file actually present in the partition directory is recorded in the manifest. If directory listing returns a file whose name is not in the manifest file list, validateManifest throws MALFORMED_HIVE_FILE_STATISTICS. This prevents using stale/incomplete statistics metadata for split planning.","triggerScenarios":"loadPartition → validateManifest where directoryLister.list returns a HiveFileInfo whose fileName is absent from manifestFileNames — i.e. a file was added/renamed in the partition directory after the manifest was generated, or the manifest is out of sync.","commonSituations":"Concurrent writes adding files to a partition after the manifest snapshot; manual file copies/renames into the partition directory (e.g. hadoop fs -put); failed or partial manifest regeneration; restoring a partition directory from backup without refreshing the manifest.","solutions":["Regenerate the manifest for the affected partition (re-run the manifest-producing job / refresh statistics) so it includes all current files.","Remove any stray files that should not be in the partition directory.","Avoid mutating partition directories while queries read the manifest; ensure writers update the manifest atomically after commits.","If manifests keep going stale, disable manifest-based statistics for the table and fall back to filesystem listing."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before reading, verify every file in the partition is in the manifest\nhdfs dfs -ls /warehouse/table/partition_dir > listed.txt\ncut -d' ' -f1 listed.txt | while read f; do grep -q \"$f\" manifest.json || echo \"MISSING: $f\"; done","typeGuard":null,"tryCatchPattern":"try {\n    return readPartition(partition);\n} catch (PrestoException e) {\n    if (\"MALFORMED_HIVE_FILE_STATISTICS\".equals(e.getErrorCode().getName())) {\n        regenerateManifest(partition);\n        return readPartition(partition);\n    }\n    throw e;\n}","preventionTips":["Never copy/rename files into partition directories manually; go through the write pipeline.","Regenerate manifests after any out-of-band partition modification.","Block writes to partitions while manifest-based reads are active, or make commits atomic."],"tags":["hive","manifest","file-statistics","metadata-inconsistency"],"backgroundTag":"manifest-out-of-sync","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}