apache/flink · error · IOException

Failed to initialize GoogleHadoopFileSystem

Error message

Failed to initialize GoogleHadoopFileSystem

What it means

Thrown by GSFileSystemFactory.create(URI) when GoogleHadoopFileSystem.initialize(fsUri, hadoopConfig) fails with an IOException. This is the entry point for the gs:// filesystem plugin, so the wrapped failure usually means the Google Cloud Storage credentials or configuration supplied to the plugin are invalid — the real reason is in the cause chain.

Source

Thrown at flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemFactory.java:156

    }

    @Override
    public String getScheme() {
        return SCHEME;
    }

    @Override
    public FileSystem create(URI fsUri) throws IOException {
        LOGGER.info("Creating GSFileSystem for uri {} with options {}", fsUri, fileSystemOptions);

        Preconditions.checkNotNull(fsUri);

        // create the Google Hadoop file system
        GoogleHadoopFileSystem googleHadoopFileSystem = new GoogleHadoopFileSystem();
        try {
            googleHadoopFileSystem.initialize(fsUri, hadoopConfig);
        } catch (IOException ex) {
            throw new IOException("Failed to initialize GoogleHadoopFileSystem", ex);
        }

        // create the file system
        return new GSFileSystem(googleHadoopFileSystem, storage, fileSystemOptions);
    }

    @VisibleForTesting
    Storage getStorage() {
        return storage;
    }

    /** Config context implementation used at runtime. */
    private static class RuntimeConfigContext implements ConfigUtils.ConfigContext {

        @Override
        public Optional<String> getenv(String name) {
            return Optional.ofNullable(System.getenv(name));
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Read the cause exception — it carries the actual GoogleHadoopFileSystem error (auth, project, endpoint)
2. Verify the credential config: fs.gs.auth.service.account.json.keyfile exists, is valid JSON, and the account has roles/storage.objectAdmin on the bucket
3. Test the same key file with 'gsutil -o Credentials:...' or gcloud storage ls to isolate Flink from GCS
4. Ensure gs:// URIs include the bucket (gs://bucket/...) and the plugin jar (flink-gs-fs-hadoop) is in Flink's plugins directory

// before first gs:// access, sanity-check the credential config
String keyfile = hadoopConf.get("fs.gs.auth.service.account.json.keyfile");
if (keyfile != null && !new java.io.File(keyfile).canRead()) {
    throw new IllegalStateException("GCS key file unreadable: " + keyfile);
}

try {
    FileSystem.get(new URI("gs://bucket/"), hadoopConf);
} catch (java.io.IOException e) {
    // unwrap cause: usually auth/project config failure from GoogleHadoopFileSystem
    log.error("GS filesystem init failed", e.getCause());
}

When it happens

Trigger: First access of a gs:// path triggers plugin filesystem creation; initialize fails on bad service-account credentials (malformed JSON, wrong key), missing GCS project/bucket config, or an inaccessible GCS endpoint.

Common situations: Wrong value for gs.credentials.* options in flink-conf.yaml (e.g. fs.gs.auth.service.account.json.keyfile pointing at a missing file); service account without storage permissions; copying configs between environments without moving the key file.

Related errors

• URL is invalid. This should not happen.
• The program's entry point class '{className}' threw an error
• The program's entry point class '{className}' caused an exce
• An I/O error occurred while creating temporary file to extra
• Exception encountered during finding the flink-python jar. T

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d4ee0506abd2b938. Report an issue: GitHub.