{"record":{"id":"83c9ae4867ae0b09","repo":"prestodb/presto","slug":"hive-invalid-bucket-files-83c9ae","errorCode":"HIVE_INVALID_BUCKET_FILES","errorMessage":"Hive table '%s' is corrupt. Found sub-directory in bucket directory for partition: %s","messagePattern":"Hive table '(.+?)' is corrupt\\. Found sub-directory in bucket directory for partition: (.+?)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/StoragePartitionLoader.java","lineNumber":570,"sourceCode":"        int tableBucketCount = bucketSplitInfo.getTableBucketCount();\n        int partitionBucketCount = bucketConversion.map(HiveSplit.BucketConversion::getPartitionBucketCount).orElse(tableBucketCount);\n\n        checkState(readBucketCount <= tableBucketCount, \"readBucketCount(%s) should be less than or equal to tableBucketCount(%s)\", readBucketCount, tableBucketCount);\n\n        // list all files in the partition\n        List<HiveFileInfo> fileInfos = new ArrayList<>(partitionBucketCount);\n        try {\n            Iterators.addAll(fileInfos, directoryLister.list(fileSystem, table, path, partition, namenodeStats, new HiveDirectoryContext(\n                    FAIL,\n                    isUseListDirectoryCache(session),\n                    isSkipEmptyFilesEnabled(session),\n                    hdfsContext.getIdentity(),\n                    buildDirectoryContextProperties(session),\n                    session.getRuntimeStats())));\n        }\n        catch (HiveFileIterator.NestedDirectoryNotAllowedException e) {\n            // Fail here to be on the safe side. This seems to be the same as what Hive does\n            throw new PrestoException(\n                    HIVE_INVALID_BUCKET_FILES,\n                    format(\"Hive table '%s' is corrupt. Found sub-directory in bucket directory for partition: %s\",\n                            table.getSchemaTableName(),\n                            partitionName));\n        }\n\n        ListMultimap<Integer, HiveFileInfo> bucketToFileInfo = computeBucketToFileInfoMapping(fileInfos, partitionBucketCount, partitionName);\n\n        // convert files internal splits\n        return convertFilesToInternalSplits(bucketSplitInfo, bucketConversion, bucketToFileInfo, splitFactory, splittable);\n    }\n\n    private ListMultimap<Integer, HiveFileInfo> computeBucketToFileInfoMapping(List<HiveFileInfo> fileInfos,\n            int partitionBucketCount,\n            String partitionName)\n    {\n        ListMultimap<Integer, HiveFileInfo> bucketToFileInfo = ArrayListMultimap.create();\n","sourceCodeStart":552,"sourceCodeEnd":588,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/StoragePartitionLoader.java#L552-L588","documentation":"When reading bucketed partitions, Presto iterates each bucket directory expecting only data files. A nested directory inside a bucket directory means the layout does not conform to Hive bucketing rules, so Presto fails fast with HIVE_INVALID_BUCKET_FILES to avoid silently returning wrong results. The catch is triggered by HiveFileIterator.NestedDirectoryNotAllowedException.","triggerScenarios":"getBucketedSplits() iterates a bucket directory and encounters a sub-directory (not a file), e.g. from _temporary leftovers, .ds_store-style dirs, or a second-level partition written inside a bucket folder.","commonSituations":"Failed/partial Hive jobs leaving _temporary directories inside bucket paths; writers that nested partition directories under bucket directories; manual data placement or rsync copying extra directories; union/multi-insert jobs writing intermediate dirs.","solutions":["List each bucket directory and remove or relocate the offending sub-directory (especially _temporary leftovers): hdfs dfs -rm -r /path/partition/bucket=0/_temporary","Re-run or clean up the failed job that produced the nested directories","Rewrite the partition data with INSERT OVERWRITE so the directory contains only flat bucket files","If nested dirs are legitimate new partitions, restructure the table so partitioning happens above the bucket level, not below it"],"exampleFix":"# before: corrupt layout\n# /table/partition=ds=1/bucket=0/_temporary/0/...\n# after\nhdfs dfs -rm -r /table/partition=ds=1/bucket=0/_temporary","handlingStrategy":"validation","validationCode":"// validate bucket dirs contain no sub-directories before querying\nfor (Path bucketDir : bucketDirs) {\n    for (FileStatus s : fs.listStatus(bucketDir)) {\n        if (s.isDirectory() && !s.getPath().getName().startsWith(\"_\") && !s.getPath().getName().startsWith(\".\")) {\n            throw new IllegalStateException(\"Sub-directory in bucket dir: \" + s.getPath());\n        }\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    connector.splitManager().getSplits(...);\n} catch (PrestoException e) {\n    if (HIVE_INVALID_BUCKET_FILES.equals(e.getErrorCode().getName())) {\n        // clean the nested directory, e.g. _temporary leftovers, then retry\n        cleanupBucketDirectories(partitionPath);\n    }\n    throw e;\n}","preventionTips":["After failed jobs, purge _temporary directories under partition paths","Never nest further partition directories inside bucket directories","Audit bucketed partitions after manual data copies (distcp, rsync)","Use INSERT OVERWRITE rather than manual placement to regenerate partitions"],"tags":["hive","bucketing","corrupt-data","directory-layout"],"backgroundTag":"invalid-bucket-file-layout","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}