apache/seatunnel · error · IMapStorageException

load all data error

Error message

load all data error

What it means

loadAll() reads the full WAL data tree under businessRootPath via WALReader.loadAllData; any IOException during reading (missing directory, corrupt file, NameNode/network failure) is rethrown as this IMapStorageException. The in-memory map cannot be fully rebuilt.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java:240

                        IMapFileData data = buildDeleteIMapFileData(key);
                        long requestId = sendToDisruptorQueue(data, WALEventType.APPEND);
                        walDisruptor.tryAppendPublish(data, requestId);
                        requestMap.put(requestId, data);
                    } catch (IOException e) {
                        log.error("parse to IMapFileData error", e);
                        failures.add(key);
                    }
                });
        return batchQueryExecuteFailsStatus(requestMap, failures);
    }

    @Override
    public Map<Object, Object> loadAll() {
        try {
            WALReader reader = new WALReader(fs, fileConfiguration, serializer);
            return reader.loadAllData(new Path(businessRootPath), new HashSet<>());
        } catch (IOException e) {
            throw new IMapStorageException("load all data error", e);
        }
    }

    @Override
    public Set<Object> loadAllKeys() {
        try {
            WALReader reader = new WALReader(fs, fileConfiguration, serializer);
            return reader.loadAllKeys(new Path(businessRootPath));
        } catch (IOException e) {
            throw new IMapStorageException(
                    e, "load all keys error parent path is {}", e, businessRootPath);
        }
    }

    @Override
    public void destroy(boolean deleteAllFileFlag) {
        log.info(
                "start destroy IMapFileStorage, businessName is {}, cluster name is {}",

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check that businessRootPath exists and is readable (hdfs dfs -ls <path>).
2. Verify WAL files are not truncated/corrupt; remove or archive damaged segments after stopping the cluster.
3. Check NameNode/datanode health and network connectivity to HDFS.
4. Ensure the running user has read permission on the storage directory.
5. Retry after transient HDFS failures; if data is disposable, call destroy(true) and re-initialize.

Example fix

// before
try {
    return reader.loadAllData(new Path(businessRootPath), new HashSet<>());
} catch (IOException e) {
    throw new IMapStorageException("load all data error", e);
}
// after
try {
    return reader.loadAllData(new Path(businessRootPath), new HashSet<>());
} catch (IOException e) {
    throw new IMapStorageException(e, "load all data error for path {}", businessRootPath);
}

org.apache.hadoop.fs.Path root = new org.apache.hadoop.fs.Path(businessRootPath);
boolean readable = fs.exists(root) && fs.getFileStatus(root).getPermission().getUserAction().implies(org.apache.hadoop.fs.permission.FsAction.READ);

try { Map<Object,Object> data = storage.loadAll(); } catch (IMapStorageException e) { log.error("loadAll failed for {}", businessRootPath, e); /* retry with backoff or re-init storage */ }

When it happens

Trigger: Calling IMapFileStorage.loadAll() when businessRootPath does not exist or was deleted, WAL segment files are corrupted, HDFS is intermittently unreachable, or permission to read the WAL directory is missing.

Common situations: Manual cleanup of the seatunnel storage directory while a cluster references it; HDFS under replication/decommission causing read failures; disk full on datanodes; restoring storage to a different cluster with a different namespace.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to list files from names${path}
• Failed to get file system
• load all keys error parent path is ${businessRootPath}
• FILE_LIST_GET_FAILED
• Failed during target_bulk_listing for path=%s

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/ea3cf1f501588d23. Report an issue: GitHub.