prestodb/presto · error · PrestoException

HIVE_INVALID_BUCKET_FILES

HIVE_INVALID_BUCKET_FILES

Error message

Hive table '%s' is corrupt. Found sub-directory in bucket directory for partition: %s

What it means

When reading bucketed partitions, Presto iterates each bucket directory expecting only data files. A nested directory inside a bucket directory means the layout does not conform to Hive bucketing rules, so Presto fails fast with HIVE_INVALID_BUCKET_FILES to avoid silently returning wrong results. The catch is triggered by HiveFileIterator.NestedDirectoryNotAllowedException.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/StoragePartitionLoader.java:570

        int tableBucketCount = bucketSplitInfo.getTableBucketCount();
        int partitionBucketCount = bucketConversion.map(HiveSplit.BucketConversion::getPartitionBucketCount).orElse(tableBucketCount);

        checkState(readBucketCount <= tableBucketCount, "readBucketCount(%s) should be less than or equal to tableBucketCount(%s)", readBucketCount, tableBucketCount);

        // list all files in the partition
        List<HiveFileInfo> fileInfos = new ArrayList<>(partitionBucketCount);
        try {
            Iterators.addAll(fileInfos, directoryLister.list(fileSystem, table, path, partition, namenodeStats, new HiveDirectoryContext(
                    FAIL,
                    isUseListDirectoryCache(session),
                    isSkipEmptyFilesEnabled(session),
                    hdfsContext.getIdentity(),
                    buildDirectoryContextProperties(session),
                    session.getRuntimeStats())));
        }
        catch (HiveFileIterator.NestedDirectoryNotAllowedException e) {
            // Fail here to be on the safe side. This seems to be the same as what Hive does
            throw new PrestoException(
                    HIVE_INVALID_BUCKET_FILES,
                    format("Hive table '%s' is corrupt. Found sub-directory in bucket directory for partition: %s",
                            table.getSchemaTableName(),
                            partitionName));
        }

        ListMultimap<Integer, HiveFileInfo> bucketToFileInfo = computeBucketToFileInfoMapping(fileInfos, partitionBucketCount, partitionName);

        // convert files internal splits
        return convertFilesToInternalSplits(bucketSplitInfo, bucketConversion, bucketToFileInfo, splitFactory, splittable);
    }

    private ListMultimap<Integer, HiveFileInfo> computeBucketToFileInfoMapping(List<HiveFileInfo> fileInfos,
            int partitionBucketCount,
            String partitionName)
    {
        ListMultimap<Integer, HiveFileInfo> bucketToFileInfo = ArrayListMultimap.create();

View on GitHub (pinned to 55bb57d202)

Solutions

List each bucket directory and remove or relocate the offending sub-directory (especially _temporary leftovers): hdfs dfs -rm -r /path/partition/bucket=0/_temporary
Re-run or clean up the failed job that produced the nested directories
Rewrite the partition data with INSERT OVERWRITE so the directory contains only flat bucket files
If nested dirs are legitimate new partitions, restructure the table so partitioning happens above the bucket level, not below it

Example fix

# before: corrupt layout
# /table/partition=ds=1/bucket=0/_temporary/0/...
# after
hdfs dfs -rm -r /table/partition=ds=1/bucket=0/_temporary

Defensive patterns

Strategy: validation

Validate before calling

// validate bucket dirs contain no sub-directories before querying
for (Path bucketDir : bucketDirs) {
    for (FileStatus s : fs.listStatus(bucketDir)) {
        if (s.isDirectory() && !s.getPath().getName().startsWith("_") && !s.getPath().getName().startsWith(".")) {
            throw new IllegalStateException("Sub-directory in bucket dir: " + s.getPath());
        }
    }
}

Try / catch

try {
    connector.splitManager().getSplits(...);
} catch (PrestoException e) {
    if (HIVE_INVALID_BUCKET_FILES.equals(e.getErrorCode().getName())) {
        // clean the nested directory, e.g. _temporary leftovers, then retry
        cleanupBucketDirectories(partitionPath);
    }
    throw e;
}

Prevention

After failed jobs, purge _temporary directories under partition paths
Never nest further partition directories inside bucket directories
Audit bucketed partitions after manual data copies (distcp, rsync)
Use INSERT OVERWRITE rather than manual placement to regenerate partitions

When it happens

Trigger: getBucketedSplits() iterates a bucket directory and encounters a sub-directory (not a file), e.g. from _temporary leftovers, .ds_store-style dirs, or a second-level partition written inside a bucket folder.

Common situations: Failed/partial Hive jobs leaving _temporary directories inside bucket paths; writers that nested partition directories under bucket directories; manual data placement or rsync copying extra directories; union/multi-insert jobs writing intermediate dirs.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/83c9ae4867ae0b09. Report an issue: GitHub.