apache/dolphinscheduler · warning · IOException

Close HadoopUtils instance failed

Error message

Close HadoopUtils instance failed

What it means

HdfsStorageOperator.close closes the underlying Hadoop FileSystem handle. If fs.close() throws IOException, it logs and rethrows an IOException wrapping the original, so callers closing the operator (e.g., in try-with-resources cleanup) are notified the HDFS connection was not cleanly released.

Source

Thrown at dolphinscheduler-storage-plugin/dolphinscheduler-storage-hdfs/src/main/java/org/apache/dolphinscheduler/plugin/storage/hdfs/HdfsStorageOperator.java:279

        return result;
    }

    @SneakyThrows
    @Override
    public StorageEntity getStorageEntity(String resourceAbsolutePath) {
        exceptionIfPathEmpty(resourceAbsolutePath);
        FileStatus fileStatus = fs.getFileStatus(new Path(resourceAbsolutePath));
        return transformFileStatusToResourceMetadata(fileStatus);
    }

    @Override
    public void close() throws IOException {
        if (fs != null) {
            try {
                fs.close();
            } catch (IOException e) {
                log.error("Close HadoopUtils instance failed", e);
                throw new IOException("Close HadoopUtils instance failed", e);
            }
        }
    }

    private StorageEntity transformFileStatusToResourceMetadata(FileStatus fileStatus) {
        Path fileStatusPath = fileStatus.getPath();
        String fileAbsolutePath = fileStatusPath.toString();
        ResourceMetadata resourceMetaData = getResourceMetaData(fileAbsolutePath);
        return StorageEntity.builder()
                .fileName(fileStatusPath.getName())
                .fullName(fileAbsolutePath)
                .pfullName(resourceMetaData.getResourceParentAbsolutePath())
                .type(resourceMetaData.getResourceType())
                .isDirectory(fileStatus.isDirectory())
                .size(fileStatus.getLen())
                .relativePath(resourceMetaData.getResourceRelativePath())
                .createTime(new Date(fileStatus.getModificationTime()))
                .updateTime(new Date(fileStatus.getModificationTime()))

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Inspect the wrapped cause (getCause()) to find the underlying HDFS/RPC failure
2. Verify NameNode connectivity and cluster health; retry the operation
3. Ensure close() is only called once per operator (use try-with-resources); update Hadoop client version to match the cluster if RPC errors persist

Example fix

// before
} catch (IOException e) {
    log.error("close failed", e);
}
// after
try (HdfsStorageOperator hdfs = new HdfsStorageOperator(...)) {
    hdfs.download(src, dst);
} // IOException from close surfaces with cause intact

try {
    hdfs.close();
} catch (IOException e) {
    log.error("HDFS close failed", e.getCause()); // inspect underlying RPC/network cause
}

When it happens

Trigger: Calling close() when the HDFS client's close fails — typically broken connections to the NameNode, interrupted threads during close, or an already-faulted FileSystem instance.

Common situations: NameNode unreachable or network partitioned at shutdown; task thread interrupted during cleanup; Hadoop client/cluster version mismatch causing RPC errors on close.

Related errors

• Failed to create parent directory for destination file
• Create xlsx directory error
• generate excel error
• Update the resource file from content: {fileAbsolutePath} fa
• Download the resource file: {fileAbsolutePath} failed

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/4d352a3eaf326ab9. Report an issue: GitHub.