apache/seatunnel · warning

Error adding Hadoop resource {}, resource was not added

Error message

Error adding Hadoop resource {}, resource was not added

What it means

This warning comes from IcebergCatalogLoader.loadHadoopConfig() when adding a Hadoop configuration resource file from the configured hadoop conf directory fails with an IOException via reflection (addResourceMethod.invoke). The catalog is still created, but that particular XML/config resource is not loaded, so any settings it carries (e.g. fs defaults, kerberos settings) are missing from the Hadoop Configuration object.

Source

Thrown at seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/IcebergCatalogLoader.java:103

            Object result = configClass.getDeclaredConstructor().newInstance();
            DynMethods.BoundMethod addResourceMethod =
                    DynMethods.builder("addResource").impl(configClass, URL.class).build(result);
            DynMethods.BoundMethod setMethod =
                    DynMethods.builder("set")
                            .impl(configClass, String.class, String.class)
                            .build(result);

            //  load any config files in the specified config directory
            String hadoopConfPath = config.getHadoopConfPath();
            if (hadoopConfPath != null) {
                HADOOP_CONF_FILES.forEach(
                        confFile -> {
                            Path path = Paths.get(hadoopConfPath, confFile);
                            if (Files.exists(path)) {
                                try {
                                    addResourceMethod.invoke(path.toUri().toURL());
                                } catch (IOException e) {
                                    log.warn(
                                            "Error adding Hadoop resource {}, resource was not added",
                                            path,
                                            e);
                                }
                            }
                        });
            }
            config.getHadoopProps().forEach(setMethod::invoke);
            // kerberos authentication
            doKerberosLogin((Configuration) result);
            log.info("Hadoop config initialized: {}", configClass.getName());
            return result;
        } catch (InstantiationException
                | IllegalAccessException
                | NoSuchMethodException
                | InvocationTargetException e) {
            log.warn(
                    "Hadoop found on classpath but could not create config, proceeding without config",

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Check the file's readability/permissions for the process user and that it is a valid Hadoop XML config.
  2. Check the attached exception (logged with this warning) for the root cause; fix the file or path accordingly.
  3. Alternatively inline the needed settings (fs.defaultFS, kerberos keys) into iceberg sink/catalog options or a known-good core-site.xml and remove the problematic file.

Example fix

// before
# hadoop-conf-path points to /etc/hadoop/conf containing a broken hdfs-site.xml
// after
ls -l /etc/hadoop/conf/*.xml && xmllint --noout /etc/hadoop/conf/hdfs-site.xml  # fix or remove the broken file
Defensive patterns

Strategy: try-catch

Validate before calling

for (String f : List.of("core-site.xml", "hdfs-site.xml")) {
    Path p = Paths.get(hadoopConfPath, f);
    if (Files.exists(p) && !Files.isReadable(p)) throw new IllegalStateException("Unreadable conf: " + p);
}

Try / catch

try {
    addResourceMethod.invoke(path.toUri().toURL());
} catch (Exception e) {
    log.error("Failed to add Hadoop resource {} - required settings may be missing", path, e);
    throw new IllegalStateException("Hadoop config resource load failed", e);
}

Prevention

When it happens

Trigger: hadoop-conf-path is set and contains a conf file (e.g. core-site.xml, hdfs-site.xml); Files.exists(path) is true, but invoking Configuration.addResource(URL) on the file throws IOException during reflection.

Common situations: Unreadable files due to permissions; malformed or special files that the Hadoop resource loader cannot open via URL; network-mounted conf dirs that briefly fail; wrong file types placed in the conf directory.

Understand the failure class

Background: "Config file not found": what it means and how to fix it in docker-sync, Maven, Vagrant, Turborepo and other tools — this error's family across 60 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/b74fb973fce8ff7f. Report an issue: GitHub.