apache/iceberg · warning

Error adding Hadoop resource

Error message

Error adding Hadoop resource {}, resource was not added

What it means

CatalogUtils.loadHadoopConfig reflectively adds files from the configured hadoopConfDir to the Hadoop Configuration via conf.addResource(URL). If adding a specific resource file throws IOException, that file is skipped with this warning; the rest of the config initialization continues.

Solutions

1. Verify the hadoopConfDir path and that referenced config files exist and are readable (chmod/chown)
2. Ensure the Connect worker process has filesystem read permissions for the config directory
3. Check the file is a valid Hadoop XML config; corrupt files can fail resource loading
4. Mount configs correctly in the container and restart the Connect worker

Example fix

// before
"hadoop-conf-dir": "/etc/hadoop/confTypo"
// after
"hadoop-conf-dir": "/etc/hadoop/conf"  // contains core-site.xml, hdfs-site.xml, readable by worker

Path dir = Paths.get(hadoopConfDir);
if (!Files.isDirectory(dir) || !Files.isReadable(dir)) throw new IllegalStateException("hadoop-conf-dir unreadable: " + dir);
try (var s = Files.list(dir)) { s.forEach(p -> { if (!Files.isReadable(p)) LOG.warn("unreadable config file: " + p); }); }

When it happens

Trigger: hadoopConfDir contains a file listed to be added that cannot be read/loaded — unreadable file permissions, malformed URL conversion, file disappearing between Files.exists check and invoke.

Common situations: Kubernetes mounts partially populated; wrong hadoop-conf-dir path; filename casing not matching hadoop-core-site.xml/hdfs-site.xml conventions; restrictive container filesystems.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Hadoop found on classpath but could not create config…
• Configuration was not serialized on purpose but was not set…
• Failed to load Configuration after loading Configurable
• getConf is not implemented
• Hadoop user is null, defaulting to user.name

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1aab4fa5d833ac38. Report an issue: GitHub.

Appendix: source

Thrown at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CatalogUtils.java:79

    try {
      Object result = DynConstructors.builder().hiddenImpl(configClass).build().newInstance();
      BoundMethod addResourceMethod =
          DynMethods.builder("addResource").impl(configClass, URL.class).build(result);
      BoundMethod setMethod =
          DynMethods.builder("set").impl(configClass, String.class, String.class).build(result);

      //  load any config files in the specified config directory
      String hadoopConfDir = config.hadoopConfDir();
      if (hadoopConfDir != null) {
        HADOOP_CONF_FILES.forEach(
            confFile -> {
              Path path = Paths.get(hadoopConfDir, confFile);
              if (Files.exists(path)) {
                try {
                  addResourceMethod.invoke(path.toUri().toURL());
                } catch (IOException e) {
                  LOG.warn("Error adding Hadoop resource {}, resource was not added", path, e);
                }
              }
            });
      }

      // set any Hadoop properties specified in the sink config
      config.hadoopProps().forEach(setMethod::invoke);

      LOG.info("Hadoop config initialized: {}", configClass.getName());
      return result;
    } catch (Exception e) {
      LOG.warn(
          "Hadoop found on classpath but could not create config, proceeding without config", e);
    }
    return null;
  }

  private CatalogUtils() {}

View on GitHub (pinned to 86d9c8fc54)