apache/hadoop · error · IOException

Native output collector cannot be loaded;

Error message

Native output collector cannot be loaded;

What it means

After all checks pass, init builds a TaskContext and calls NativeCollectorOnlyHandler.create(taskContext), which sets up the native collector (buffers, handler objects, JNI calls). If that throws an IOException, init re-throws it as IOException('Native output collector cannot be loaded;') with the original exception chained as the cause. On success you see 'Native output collector can be successfully enabled!'.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/java/org/apache/hadoop/mapred/nativetask/NativeMapOutputCollectorDelegator.java:165

    } else {
      String message = "NativeRuntime cannot be loaded, please check that " +
        "libnativetask.so is in hadoop library dir";
      LOG.error(message);
      throw new InvalidJobConfException(message);
    }

    this.handler = null;
    try {
      final Class<K> oKClass = (Class<K>) job.getMapOutputKeyClass();
      final Class<K> oVClass = (Class<K>) job.getMapOutputValueClass();
      final TaskAttemptID id = context.getMapTask().getTaskID();
      final TaskContext taskContext = new TaskContext(job, null, null, oKClass, oVClass,
          context.getReporter(), id);
      handler = NativeCollectorOnlyHandler.create(taskContext);
    } catch (final IOException e) {
      String message = "Native output collector cannot be loaded;";
      LOG.error(message);
      throw new IOException(message, e);
    }

    LOG.info("Native output collector can be successfully enabled!");
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. Read the chained cause (getCause()) in the task log - it carries the actual failure; this outer message is only a wrapper
2. Verify jar/so version alignment and rerun 'hadoop checknative'
3. Disable the native collector for the job (unset mapreduce.job.map.output.collector.class) to restore throughput while investigating

try {
  collector.init(context);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Native output collector cannot be loaded")) {
    Throwable root = e.getCause(); // actual failure is chained here
    LOG.error("native collector setup failed", root);
    // decide: fix env per root cause, or rerun with Java collector
  }
  throw e;
}

When it happens

Trigger: Any IOException inside NativeCollectorOnlyHandler.create during map-task startup: native object creation failing at handler setup, buffer allocation problems, or downstream configuration errors - with the native library loaded and all job-conf gates already passed.

Common situations: Memory-constrained nodes failing native buffer allocation; partially working native installs where earlier checks pass but specific handler creation fails; version drift between Java protocol constants and the C++ library.

Related errors

• JNI_GetCreatedJavaVMs failed with error: %d
• Native object create failed, class: {}
• Cannot find serializer for {}
• Native runtime library not loaded
• output buffer not set

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/83bc7977c3267a1f. Report an issue: GitHub.