prestodb/presto · error · NestedDirectoryNotAllowedException

Nested sub-directories are not allowed

Error message

Nested sub-directories are not allowed

What it means

HiveFileIterator walks a partition directory and, for each nested directory it encounters, applies a configured directory-listing strategy. When the strategy is FAIL it throws NestedDirectoryNotAllowedException, whose message is 'Nested sub-directories are not allowed'. The iterator enforces a flat file layout (typical for Hive partitions) and refuses to recurse silently.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/util/HiveFileIterator.java:85

        while (true) {
            while (remoteIterator.hasNext()) {
                HiveFileInfo fileInfo = getLocatedFileStatus(remoteIterator);

                // Ignore hidden files and directories. Hive ignores files starting with _ and . as well.
                String fileName = fileInfo.getFileName();
                if (fileName.startsWith("_") || fileName.startsWith(".") || (fileInfo.getLength() == 0 && skipEmptyFiles)) {
                    continue;
                }

                if (fileInfo.isDirectory()) {
                    switch (nestedDirectoryPolicy) {
                        case IGNORED:
                            continue;
                        case RECURSE:
                            paths.add(new Path(fileInfo.getPath()));
                            continue;
                        case FAIL:
                            throw new NestedDirectoryNotAllowedException();
                    }
                }

                return fileInfo;
            }

            if (paths.isEmpty()) {
                return endOfData();
            }
            remoteIterator = getLocatedFileStatusRemoteIterator(paths.removeFirst());
        }
    }

    private Iterator<HiveFileInfo> getLocatedFileStatusRemoteIterator(Path path)
    {
        try (TimeStat.BlockTimer ignored = namenodeStats.getListLocatedStatus().time()) {
            return new FileStatusIterator(path, listDirectoryOperation, namenodeStats);
        }

View on GitHub (pinned to 55bb57d202)

Solutions

1. Flatten the partition directory so it contains only data files (move files up, remove sub-directories)
2. Point the partition/table LOCATION at the correct leaf directory
3. If recursion is intended, use the API accepting RECURSE NestedDirectoryPolicy (e.g. getTableFilesRecursive)
4. Fix the upstream job that created nested sub-directories inside the partition
5. Check for hidden/temp folders (e.g. _temporary) and clean them

Example fix

// before
hdfs://.../table/part=2026-01-01/batch1/data.orc  (nested dir -> throws)
// after
hdfs://.../table/part=2026-01-01/data.orc

// before listing, assert the partition path is a flat directory
for (FileStatus f : fileSystem.listStatus(partitionPath)) {
    if (f.isDirectory() && !f.getPath().getName().startsWith("_")) {
        throw new PrestoException(HIVE_BAD_DATA,
            "Partition contains sub-directory: " + f.getPath());
    }
}

try (HiveFileIterator it = new HiveFileIterator(path, fs, nnStats, policy, ...)) {
    while (it.hasNext()) { files.add(it.next()); }
} catch (PrestoException e) {
    if (e.getMessage() != null && e.getMessage().contains("Nested sub-directories")) {
        LOG.error("Flat layout violated under " + path);
    }
    throw e;
}

When it happens

Trigger: Listing files under a partition path that contains sub-directories while the iterator's NestedDirectoryPolicy is FAIL (the default for partitioned tables), e.g. via getPartitionFiles/recursive listing APIs.

Common situations: Writer jobs creating sub-directories inside a partition (e.g. part=2026-01-01/extra/); compaction or ETL tools leaving nested folders; misconfigured partition location pointing at a parent directory that contains other partitions.

Related errors

• HIVE_FILESYSTEM_ERROR
• HIVE_FILESYSTEM_ERROR
• HIVE_PARTITION_SCHEMA_MISMATCH
• HIVE_PARTITION_DROPPED_DURING_QUERY
• NOT_SUPPORTED

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/99941622db9937cd. Report an issue: GitHub.