apache/flink · error · RuntimeException

DistributedCache supports only local files for Collection En

Error message

DistributedCache supports only local files for Collection Environments

What it means

Thrown by the CompletedFuture inner class constructor in CollectionExecutor when resolving a DistributedCache file path fails. The constructor tries to resolve the path via FileSystem.getUnguardedFileSystem() and cast it to LocalFileSystem; if the path refers to a non-local scheme (e.g., hdfs://, s3://), the cast or resolution throws and the catch block wraps it as a RuntimeException. Collection environments only support local files because they execute in-process without a distributed filesystem.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/operators/CollectionExecutor.java:634

        public <T extends Value> T getPreviousIterationAggregate(String name) {
            return (T) previousAggregates.get(name);
        }
    }

    private static final class CompletedFuture implements Future<Path> {

        private final Path result;

        public CompletedFuture(Path entry) {
            try {
                LocalFileSystem fs =
                        (LocalFileSystem) FileSystem.getUnguardedFileSystem(entry.toUri());
                result =
                        entry.isAbsolute()
                                ? new Path(entry.toUri().getPath())
                                : new Path(fs.getWorkingDirectory(), entry);
            } catch (Exception e) {
                throw new RuntimeException(
                        "DistributedCache supports only local files for Collection Environments");
            }
        }

        @Override
        public boolean cancel(boolean mayInterruptIfRunning) {
            return false;
        }

        @Override
        public boolean isCancelled() {
            return false;
        }

        @Override
        public boolean isDone() {
            return true;
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Use only local file paths (file:// or relative paths) when registering cached files with a CollectionEnvironment.
2. Switch to LocalEnvironment or a mini-cluster if remote file access is needed.
3. Download the remote file to a local temporary directory before registering it as a cached file for local testing.

Example fix

// before — remote path with collection environment
env = new CollectionEnvironment();
env.registerCachedFile("hdfs://namenode/cache/data.txt", "data");
// after — local path for collection environment
env = new CollectionEnvironment();
env.registerCachedFile("/tmp/cache/data.txt", "data");

// Validate cached file scheme before registering with CollectionEnvironment
Path path = new Path(filePath);
String scheme = path.toUri().getScheme();
if (scheme != null && !scheme.equals("file")) {
    throw new IllegalArgumentException(
        "CollectionEnvironment only supports local (file://) cached files, got: " + scheme);
}
env.registerCachedFile(filePath, name);

When it happens

Trigger: Registering a DistributedCache file with a non-local URI scheme (hdfs://, s3://, gs://, etc.) and then executing the job with a CollectionEnvironment. The constructor catches any Exception from FileSystem.getUnguardedFileSystem() or the LocalFileSystem cast and throws this generic message.

Common situations: A developer registers a cached file via env.registerCachedFile("hdfs://namenode/path", "name") and runs locally with CollectionEnvironment. The collection executor cannot access remote filesystems, so it rejects the file. Also triggered if the local file path is invalid or unreadable (though the message would be the same due to the broad catch block).

Related errors

• Failed to get the FileSystem of artifact {artifactFilePath}.
• An error occurred while copying the file.
• Cannot execute operator {}
• The data sink {} has no input.
• The unary operation {} has no input.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a51a575a4b93f194. Report an issue: GitHub.