{"record":{"id":"a80ff4432affc762","repo":"apache/hadoop","slug":"nativeruntime-cannot-be-loaded-please-check-that","errorCode":null,"errorMessage":"NativeRuntime cannot be loaded, please check that libnativetask.so is in hadoop library dir","messagePattern":"NativeRuntime cannot be loaded, please check that libnativetask\\.so is in hadoop library dir","errorType":"exception","errorClass":"InvalidJobConfException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java","lineNumber":151,"sourceCode":"        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);\n        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {\n          String message = \"Native output collector doesn't support compression codec \" + codec;\n          LOG.error(message);\n          throw new InvalidJobConfException(message);\n        }\n      }\n      NativeRuntime.configure(job);\n\n      final long updateInterval = job.getLong(Constants.NATIVE_STATUS_UPDATE_INTERVAL,\n          Constants.NATIVE_STATUS_UPDATE_INTERVAL_DEFVAL);\n      updater = new StatusReportChecker(context.getReporter(), updateInterval);\n      updater.start();\n\n    } else {\n      String message = \"NativeRuntime cannot be loaded, please check that \" +\n        \"libnativetask.so is in hadoop library dir\";\n      LOG.error(message);\n      throw new InvalidJobConfException(message);\n    }\n\n    this.handler = null;\n    try {\n      final Class<K> oKClass = (Class<K>) job.getMapOutputKeyClass();\n      final Class<K> oVClass = (Class<K>) job.getMapOutputValueClass();\n      final TaskAttemptID id = context.getMapTask().getTaskID();\n      final TaskContext taskContext = new TaskContext(job, null, null, oKClass, oVClass,\n          context.getReporter(), id);\n      handler = NativeCollectorOnlyHandler.create(taskContext);\n    } catch (final IOException e) {\n      String message = \"Native output collector cannot be loaded;\";\n      LOG.error(message);\n      throw new IOException(message, e);\n    }\n\n    LOG.info(\"Native output collector can be successfully enabled!\");\n  }","sourceCodeStart":133,"sourceCodeEnd":169,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java#L133-L169","documentation":"init's final gate is NativeRuntime.isNativeLibraryLoaded(); false means the static initializer's System.loadLibrary('nativetask') failed (missing or incompatible libnativetask.so). The earlier log line 'Failed to load nativetask JNI library with error: ...' plus java.library.path and LD_LIBRARY_PATH are printed at load time. init throws InvalidJobConfException telling the operator to check that libnativetask.so is in the Hadoop library dir.","triggerScenarios":"Enabling the native collector on a node where libnativetask.so is absent (minimal installs, containers without native libs mounted) or incompatible with the JVM/arch, so isNativeLibraryLoaded() returns false at init.","commonSituations":"HADOOP_HOME installs missing lib/native; platform/arch mismatch (e.g. no .so for the OS arch); LD_LIBRARY_PATH or -Djava.library.path not covering the native dir; slim Docker images for NodeManagers built without the native tarball.","solutions":["Run 'hadoop checknative -a' on the affected node to confirm libnativetask.so presence and loadability","Install/extract the matching Hadoop native library dir and set HADOOP_OPTS/-Djava.library.path or LD_LIBRARY_PATH to include it","If native acceleration is not a requirement, unset mapreduce.job.map.output.collector.class so the default Java collector runs"],"exampleFix":"# before: native dir not on the library path, job enables native collector\nexport HADOOP_OPTS=\"...\"\n\n# after\nexport LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH\nexport HADOOP_OPTS=\"$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native\"\nhadoop checknative -a   # expect: NativeTask: true","handlingStrategy":"validation","validationCode":"boolean wantNative = \"org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator\"\n    .equals(jobConf.get(\"mapreduce.job.map.output.collector.class\"));\nif (wantNative && !NativeRuntime.isNativeLibraryLoaded()) {\n  jobConf.unset(\"mapreduce.job.map.output.collector.class\"); // libnativetask.so unavailable\n}","typeGuard":null,"tryCatchPattern":"try {\n  collector.init(context);\n} catch (InvalidJobConfException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"libnativetask.so\")) {\n    // environment fix required: install native libs, then retry; or stay on Java collector\n  }\n  throw e;\n}","preventionTips":["Bake lib/native into NodeManager/container images and verify with 'hadoop checknative -a'","Set java.library.path/LD_LIBRARY_PATH to $HADOOP_HOME/lib/native in env","Gate native-collector rollout on a per-node native-library health check"],"tags":["nativetask","native-library","environment","deployment"],"backgroundTag":"native-library-load-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}