prestodb/presto · error · PrestoException

HIVE_FILESYSTEM_ERROR

HIVE_FILESYSTEM_ERROR

Error message

Failed to delete partition %s files during overwrite

What it means

Thrown when the connector fails to delete existing data files while overwriting a Hive partition. During an INSERT OVERWRITE, Presto deletes the old files in the partition directory (skipping files created by the current query) before writing new data; any filesystem-level failure (HDFS/DFSClient error, permission denial, network hiccup) surfaces here wrapped in HIVE_FILESYSTEM_ERROR with the original exception as cause.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/HiveMetadata.java:2411

     *
     * @param session the ConnectorSession object
     * @param partitionPath the path of the partition from where the older files are to be deleted
     */
    private void removeNonCurrentQueryFiles(ConnectorSession session, Path partitionPath)
    {
        String queryId = session.getQueryId();
        try {
            FileSystem fileSystem = hdfsEnvironment.getFileSystem(new HdfsContext(session), partitionPath);
            RemoteIterator<LocatedFileStatus> iterator = fileSystem.listFiles(partitionPath, false);
            while (iterator.hasNext()) {
                Path file = iterator.next().getPath();
                if (!isFileCreatedByQuery(file.getName(), queryId)) {
                    fileSystem.delete(file, false);
                }
            }
        }
        catch (Exception ex) {
            throw new PrestoException(
                    HIVE_FILESYSTEM_ERROR,
                    format("Failed to delete partition %s files during overwrite", partitionPath),
                    ex);
        }
    }

    private static boolean isTempPathRequired(ConnectorSession session, Optional<HiveBucketProperty> bucketProperty, List<SortingColumn> preferredOrderingColumns)
    {
        boolean hasSortedWrite = bucketProperty.map(property -> !property.getSortedBy().isEmpty()).orElse(false) || !preferredOrderingColumns.isEmpty();
        return isSortedWriteToTempPathEnabled(session) && hasSortedWrite;
    }

    private List<String> getTargetFileNames(List<FileWriteInfo> fileWriteInfos)
    {
        return fileWriteInfos.stream()
                .map(FileWriteInfo::getTargetFileName)
                .collect(toImmutableList());
    }

View on GitHub (pinned to 55bb57d202)

Solutions

1. Check the cause exception for the underlying filesystem error (permissions, name node connectivity) and fix that first
2. Verify the querying user has WRITE/EXECUTE permission on the partition directory and its parent in HDFS
3. Check for concurrent jobs (Spark compaction, other inserts) mutating the partition and retry after they finish
4. Retry the overwrite once the transient HDFS issue is resolved
5. If files were externally moved/renamed, repair the partition location so listing and deletion succeed

Example fix

// before: blind overwrite that races with external deletes
fileSystem.delete(file, false);
// after: tolerate already-vanished files and surface a clearer error
if (fileSystem.exists(file) && !isFileCreatedByQuery(file.getName(), queryId)) {
    fileSystem.delete(file, false);
}

// before overwriting, check writability of the partition dir
FileSystem fs = partitionPath.getFileSystem(conf);
if (!fs.exists(partitionPath) || !fs.getFileStatus(partitionPath).getPermission().getUserAction().implies(FsAction.WRITE)) {
    throw new IllegalStateException("No write access to partition: " + partitionPath);
}

try {
    runInsertOverwrite(...);
} catch (PrestoException e) {
    if (e.getErrorCode().equals(HIVE_FILESYSTEM_ERROR.toErrorCode())) {
        // inspect e.getCause() for permission vs transient IO; retry transient, alert otherwise
    }
    throw e;
}

When it happens

Trigger: INSERT OVERWRITE or delete-of-partition-files path where fileSystem.delete(file, false) or directory listing throws — e.g. HDFS name node unreachable, user lacks write permission on the partition directory, file already moved/deleted by another job, or a non-recursive delete hitting a subdirectory.

Common situations: Concurrent jobs writing/compacting the same partition; HDFS permission changes after table creation; transient NameNode/DataNode outages; partition location moved to read-only storage; stale file handles after cluster maintenance.

Related errors

• HIVE_FILESYSTEM_ERROR
• HIVE_FILESYSTEM_ERROR
• HIVE_WRITER_CLOSE_ERROR
• HIVE_WRITER_OPEN_ERROR
• HIVE_BAD_DATA

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/4660c1e9fe58492c. Report an issue: GitHub.