apache/hadoop · error · InvalidJobConfException

NativeRuntime cannot be loaded, please check that libnativet

Error message

NativeRuntime cannot be loaded, please check that libnativetask.so is in hadoop library dir

What it means

init's final gate is NativeRuntime.isNativeLibraryLoaded(); false means the static initializer's System.loadLibrary('nativetask') failed (missing or incompatible libnativetask.so). The earlier log line 'Failed to load nativetask JNI library with error: ...' plus java.library.path and LD_LIBRARY_PATH are printed at load time. init throws InvalidJobConfException telling the operator to check that libnativetask.so is in the Hadoop library dir.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:151

        String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
        if (!NativeRuntime.supportsCompressionCodec(codec.getBytes(StandardCharsets.UTF_8))) {
          String message = "Native output collector doesn't support compression codec " + codec;
          LOG.error(message);
          throw new InvalidJobConfException(message);
        }
      }
      NativeRuntime.configure(job);

      final long updateInterval = job.getLong(Constants.NATIVE_STATUS_UPDATE_INTERVAL,
          Constants.NATIVE_STATUS_UPDATE_INTERVAL_DEFVAL);
      updater = new StatusReportChecker(context.getReporter(), updateInterval);
      updater.start();

    } else {
      String message = "NativeRuntime cannot be loaded, please check that " +
        "libnativetask.so is in hadoop library dir";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    this.handler = null;
    try {
      final Class<K> oKClass = (Class<K>) job.getMapOutputKeyClass();
      final Class<K> oVClass = (Class<K>) job.getMapOutputValueClass();
      final TaskAttemptID id = context.getMapTask().getTaskID();
      final TaskContext taskContext = new TaskContext(job, null, null, oKClass, oVClass,
          context.getReporter(), id);
      handler = NativeCollectorOnlyHandler.create(taskContext);
    } catch (final IOException e) {
      String message = "Native output collector cannot be loaded;";
      LOG.error(message);
      throw new IOException(message, e);
    }

    LOG.info("Native output collector can be successfully enabled!");
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hadoop checknative -a' on the affected node to confirm libnativetask.so presence and loadability
2. Install/extract the matching Hadoop native library dir and set HADOOP_OPTS/-Djava.library.path or LD_LIBRARY_PATH to include it
3. If native acceleration is not a requirement, unset mapreduce.job.map.output.collector.class so the default Java collector runs

Example fix

# before: native dir not on the library path, job enables native collector
export HADOOP_OPTS="..."

# after
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"
hadoop checknative -a   # expect: NativeTask: true

boolean wantNative = "org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator"
    .equals(jobConf.get("mapreduce.job.map.output.collector.class"));
if (wantNative && !NativeRuntime.isNativeLibraryLoaded()) {
  jobConf.unset("mapreduce.job.map.output.collector.class"); // libnativetask.so unavailable
}

try {
  collector.init(context);
} catch (InvalidJobConfException e) {
  if (e.getMessage() != null && e.getMessage().contains("libnativetask.so")) {
    // environment fix required: install native libs, then retry; or stay on Java collector
  }
  throw e;
}

When it happens

Trigger: Enabling the native collector on a node where libnativetask.so is absent (minimal installs, containers without native libs mounted) or incompatible with the JVM/arch, so isNativeLibraryLoaded() returns false at init.

Common situations: HADOOP_HOME installs missing lib/native; platform/arch mismatch (e.g. no .so for the OS arch); LD_LIBRARY_PATH or -Djava.library.path not covering the native dir; slim Docker images for NodeManagers built without the native tarball.

Related errors

• Native object create failed, class: {}
• Native runtime library not loaded
• Authentication type must be specified: simple|kerberos|<clas
• The OpenSSL native library is built without SM4 CTR support
• Fewer lines of output than expected

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a80ff4432affc762. Report an issue: GitHub.