{"record":{"id":"18f877fca0f92a39","repo":"apache/iceberg","slug":"hadoop-found-on-classpath-but-could-not-create-con","errorCode":null,"errorMessage":"Hadoop found on classpath but could not create config, proceeding without config","messagePattern":"Hadoop found on classpath but could not create config, proceeding without config","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CatalogUtils.java","lineNumber":91,"sourceCode":"            confFile -> {\n              Path path = Paths.get(hadoopConfDir, confFile);\n              if (Files.exists(path)) {\n                try {\n                  addResourceMethod.invoke(path.toUri().toURL());\n                } catch (IOException e) {\n                  LOG.warn(\"Error adding Hadoop resource {}, resource was not added\", path, e);\n                }\n              }\n            });\n      }\n\n      // set any Hadoop properties specified in the sink config\n      config.hadoopProps().forEach(setMethod::invoke);\n\n      LOG.info(\"Hadoop config initialized: {}\", configClass.getName());\n      return result;\n    } catch (Exception e) {\n      LOG.warn(\n          \"Hadoop found on classpath but could not create config, proceeding without config\", e);\n    }\n    return null;\n  }\n\n  private CatalogUtils() {}\n}\n","sourceCodeStart":73,"sourceCodeEnd":99,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/CatalogUtils.java#L73-L99","documentation":"CatalogUtils.loadHadoopConfig attempts to locate a Hadoop Configuration class and build a config object reflectively. If any step throws (class missing, instantiation failure, property-setting errors), it logs this warning and returns null, meaning the catalog proceeds with no Hadoop configuration loaded.","triggerScenarios":"Hadoop classes are on the classpath (so the code tries) but org.apache.hadoop.conf.Configuration cannot be instantiated or configured — e.g. hadoop-common present but broken/incomplete, hadoopProps containing invalid setter calls, or reflection failures.","commonSituations":"Shaded/fat jars excluding needed Hadoop classes; version conflicts between hadoop-common jars; hadoopProps keys that are not valid Configuration setters; Connect worker plugin classloader isolation.","solutions":["Add a complete, consistent hadoop-common (and hadoop-client) dependency set to the worker classpath","Move Hadoop jars out of plugin isolation paths if the Connect classloader hides them from each other","Fix invalid entries in hadoopProps in the sink config so setProperty calls succeed","Or intentionally drop Hadoop config reliance — the connector proceeds without it, using catalog-specific config"],"exampleFix":"// before\n\"hadoop.props\": { \"invalid.key.setting\": \"x\" }  // reflection fails, whole config dropped\n// after\n\"hadoop.props\": { \"fs.defaultFS\": \"hdfs://nameservice1\" }","handlingStrategy":"fallback","validationCode":"try { Class.forName(\"org.apache.hadoop.conf.Configuration\"); } catch (ClassNotFoundException e) { /* Hadoop not fully on classpath */ }","typeGuard":null,"tryCatchPattern":"try { conf = loadHadoopConfig(config); } catch (Exception e) { conf = new org.apache.hadoop.conf.Configuration(); } // or proceed without","preventionTips":["Ship a complete hadoop-common/hadoop-client set with matching versions","Keep Hadoop jars on the parent classpath, not plugin-isolated paths","Keep hadoopProps limited to valid Configuration properties","Log the resolved Hadoop config at startup to confirm it loaded"],"tags":["hadoop","kafka-connect","classpath","configuration"],"backgroundTag":"module-init-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}