prestodb/presto · error · PrestoException

MALFORMED_HIVE_FILE_STATISTICS

MALFORMED_HIVE_FILE_STATISTICS

Error message

Filename = %s not stored in manifest. Partition = %s, TableName = %s

What it means

When a partition is loaded with manifest-based file statistics, Presto validates that every file actually present in the partition directory is recorded in the manifest. If directory listing returns a file whose name is not in the manifest file list, validateManifest throws MALFORMED_HIVE_FILE_STATISTICS. This prevents using stale/incomplete statistics metadata for split planning.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/ManifestPartitionLoader.java:208

    private void validateManifest(ConnectorSession session, HivePartitionMetadata partition, Path path, List<String> manifestFileNames, List<Long> manifestFileSizes)
            throws IOException
    {
        ExtendedFileSystem fileSystem = hdfsEnvironment.getFileSystem(hdfsContext, path);
        HiveDirectoryContext hiveDirectoryContext = new HiveDirectoryContext(
                recursiveDirWalkerEnabled ? RECURSE : IGNORED,
                false,
                isSkipEmptyFilesEnabled(session),
                hdfsContext.getIdentity(),
                buildDirectoryContextProperties(session),
                session.getRuntimeStats());

        Iterator<HiveFileInfo> fileInfoIterator = directoryLister.list(fileSystem, table, path, partition.getPartition(), namenodeStats, hiveDirectoryContext);
        int fileCount = 0;
        while (fileInfoIterator.hasNext()) {
            HiveFileInfo fileInfo = fileInfoIterator.next();
            String fileName = fileInfo.getFileName();
            if (!manifestFileNames.contains(fileName)) {
                throw new PrestoException(
                        MALFORMED_HIVE_FILE_STATISTICS,
                        format("Filename = %s not stored in manifest. Partition = %s, TableName = %s",
                                fileName,
                                partition.getHivePartition().getPartitionId(),
                                table.getTableName()));
            }

            int index = manifestFileNames.indexOf(fileName);
            if (!manifestFileSizes.get(index).equals(fileInfo.getLength())) {
                throw new PrestoException(
                        MALFORMED_HIVE_FILE_STATISTICS,
                        format("FilesizeFromManifest = %s is not equal to FilesizeFromStorage = %s. File = %s, Partition = %s, TableName = %s",
                                manifestFileSizes.get(index),
                                fileInfo.getLength(),
                                fileName,
                                partition.getHivePartition().getPartitionId(),
                                table.getTableName()));
            }

View on GitHub (pinned to 55bb57d202)

Solutions

Regenerate the manifest for the affected partition (re-run the manifest-producing job / refresh statistics) so it includes all current files.
Remove any stray files that should not be in the partition directory.
Avoid mutating partition directories while queries read the manifest; ensure writers update the manifest atomically after commits.
If manifests keep going stale, disable manifest-based statistics for the table and fall back to filesystem listing.

Defensive patterns

Strategy: validation

Validate before calling

// Before reading, verify every file in the partition is in the manifest
hdfs dfs -ls /warehouse/table/partition_dir > listed.txt
cut -d' ' -f1 listed.txt | while read f; do grep -q "$f" manifest.json || echo "MISSING: $f"; done

Try / catch

try {
    return readPartition(partition);
} catch (PrestoException e) {
    if ("MALFORMED_HIVE_FILE_STATISTICS".equals(e.getErrorCode().getName())) {
        regenerateManifest(partition);
        return readPartition(partition);
    }
    throw e;
}

Prevention

Never copy/rename files into partition directories manually; go through the write pipeline.
Regenerate manifests after any out-of-band partition modification.
Block writes to partitions while manifest-based reads are active, or make commits atomic.

When it happens

Trigger: loadPartition → validateManifest where directoryLister.list returns a HiveFileInfo whose fileName is absent from manifestFileNames — i.e. a file was added/renamed in the partition directory after the manifest was generated, or the manifest is out of sync.

Common situations: Concurrent writes adding files to a partition after the manifest snapshot; manual file copies/renames into the partition directory (e.g. hadoop fs -put); failed or partial manifest regeneration; restoring a partition directory from backup without refreshing the manifest.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/88fd91a9007b1e35. Report an issue: GitHub.