apache/iceberg · warning

Hadoop found on classpath but could not create config…

Error message

Hadoop found on classpath but could not create config, proceeding without config

What it means

CatalogUtils.loadHadoopConfig attempts to locate a Hadoop Configuration class and build a config object reflectively. If any step throws (class missing, instantiation failure, property-setting errors), it logs this warning and returns null, meaning the catalog proceeds with no Hadoop configuration loaded.

Solutions

1. Add a complete, consistent hadoop-common (and hadoop-client) dependency set to the worker classpath
2. Move Hadoop jars out of plugin isolation paths if the Connect classloader hides them from each other
3. Fix invalid entries in hadoopProps in the sink config so setProperty calls succeed
4. Or intentionally drop Hadoop config reliance — the connector proceeds without it, using catalog-specific config

Example fix

// before
"hadoop.props": { "invalid.key.setting": "x" }  // reflection fails, whole config dropped
// after
"hadoop.props": { "fs.defaultFS": "hdfs://nameservice1" }

try { Class.forName("org.apache.hadoop.conf.Configuration"); } catch (ClassNotFoundException e) { /* Hadoop not fully on classpath */ }

try { conf = loadHadoopConfig(config); } catch (Exception e) { conf = new org.apache.hadoop.conf.Configuration(); } // or proceed without

When it happens

Trigger: Hadoop classes are on the classpath (so the code tries) but org.apache.hadoop.conf.Configuration cannot be instantiated or configured — e.g. hadoop-common present but broken/incomplete, hadoopProps containing invalid setter calls, or reflection failures.

Common situations: Shaded/fat jars excluding needed Hadoop classes; version conflicts between hadoop-common jars; hadoopProps keys that are not valid Configuration setters; Connect worker plugin classloader isolation.

Related errors

• Error adding Hadoop resource
• Cannot initialize TLSConfigurer implementation
• Class was not found
• Configuration was not serialized on purpose but was not set…
• Failed to load catalog

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/18f877fca0f92a39. Report an issue: GitHub.

Appendix: source

Thrown at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CatalogUtils.java:91

            confFile -> {
              Path path = Paths.get(hadoopConfDir, confFile);
              if (Files.exists(path)) {
                try {
                  addResourceMethod.invoke(path.toUri().toURL());
                } catch (IOException e) {
                  LOG.warn("Error adding Hadoop resource {}, resource was not added", path, e);
                }
              }
            });
      }

      // set any Hadoop properties specified in the sink config
      config.hadoopProps().forEach(setMethod::invoke);

      LOG.info("Hadoop config initialized: {}", configClass.getName());
      return result;
    } catch (Exception e) {
      LOG.warn(
          "Hadoop found on classpath but could not create config, proceeding without config", e);
    }
    return null;
  }

  private CatalogUtils() {}
}

View on GitHub (pinned to 86d9c8fc54)