{"record":{"id":"99941622db9937cd","repo":"prestodb/presto","slug":"nested-sub-directories-are-not-allowed","errorCode":null,"errorMessage":"Nested sub-directories are not allowed","messagePattern":"Nested sub-directories are not allowed","errorType":"exception","errorClass":"NestedDirectoryNotAllowedException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/util/HiveFileIterator.java","lineNumber":85,"sourceCode":"        while (true) {\n            while (remoteIterator.hasNext()) {\n                HiveFileInfo fileInfo = getLocatedFileStatus(remoteIterator);\n\n                // Ignore hidden files and directories. Hive ignores files starting with _ and . as well.\n                String fileName = fileInfo.getFileName();\n                if (fileName.startsWith(\"_\") || fileName.startsWith(\".\") || (fileInfo.getLength() == 0 && skipEmptyFiles)) {\n                    continue;\n                }\n\n                if (fileInfo.isDirectory()) {\n                    switch (nestedDirectoryPolicy) {\n                        case IGNORED:\n                            continue;\n                        case RECURSE:\n                            paths.add(new Path(fileInfo.getPath()));\n                            continue;\n                        case FAIL:\n                            throw new NestedDirectoryNotAllowedException();\n                    }\n                }\n\n                return fileInfo;\n            }\n\n            if (paths.isEmpty()) {\n                return endOfData();\n            }\n            remoteIterator = getLocatedFileStatusRemoteIterator(paths.removeFirst());\n        }\n    }\n\n    private Iterator<HiveFileInfo> getLocatedFileStatusRemoteIterator(Path path)\n    {\n        try (TimeStat.BlockTimer ignored = namenodeStats.getListLocatedStatus().time()) {\n            return new FileStatusIterator(path, listDirectoryOperation, namenodeStats);\n        }","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/util/HiveFileIterator.java#L67-L103","documentation":"HiveFileIterator walks a partition directory and, for each nested directory it encounters, applies a configured directory-listing strategy. When the strategy is FAIL it throws NestedDirectoryNotAllowedException, whose message is 'Nested sub-directories are not allowed'. The iterator enforces a flat file layout (typical for Hive partitions) and refuses to recurse silently.","triggerScenarios":"Listing files under a partition path that contains sub-directories while the iterator's NestedDirectoryPolicy is FAIL (the default for partitioned tables), e.g. via getPartitionFiles/recursive listing APIs.","commonSituations":"Writer jobs creating sub-directories inside a partition (e.g. part=2026-01-01/extra/); compaction or ETL tools leaving nested folders; misconfigured partition location pointing at a parent directory that contains other partitions.","solutions":["Flatten the partition directory so it contains only data files (move files up, remove sub-directories)","Point the partition/table LOCATION at the correct leaf directory","If recursion is intended, use the API accepting RECURSE NestedDirectoryPolicy (e.g. getTableFilesRecursive)","Fix the upstream job that created nested sub-directories inside the partition","Check for hidden/temp folders (e.g. _temporary) and clean them"],"exampleFix":"// before\nhdfs://.../table/part=2026-01-01/batch1/data.orc  (nested dir -> throws)\n// after\nhdfs://.../table/part=2026-01-01/data.orc","handlingStrategy":"validation","validationCode":"// before listing, assert the partition path is a flat directory\nfor (FileStatus f : fileSystem.listStatus(partitionPath)) {\n    if (f.isDirectory() && !f.getPath().getName().startsWith(\"_\")) {\n        throw new PrestoException(HIVE_BAD_DATA,\n            \"Partition contains sub-directory: \" + f.getPath());\n    }\n}","typeGuard":null,"tryCatchPattern":"try (HiveFileIterator it = new HiveFileIterator(path, fs, nnStats, policy, ...)) {\n    while (it.hasNext()) { files.add(it.next()); }\n} catch (PrestoException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Nested sub-directories\")) {\n        LOG.error(\"Flat layout violated under \" + path);\n    }\n    throw e;\n}","preventionTips":["Enforce flat file layout in all writer/ETL jobs writing into partitions","Clean _temporary / staged folders before queries read a partition","Use RECURSE policy explicitly when nested layouts are intentional","Point table/partition LOCATION at leaf directories only"],"tags":["hive","filesystem","directory-structure","partition"],"backgroundTag":"nested-directories-not-allowed","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}