apache/hadoop · error · IOException

runtime classes not found: {}

Error message

runtime classes not found: {}

What it means

StreamJob must ship the job submitter's runtime classes (the hadoop-streaming jar) to the cluster so tasks can load streaming code. It first honors the property stream.shipped.hadoopstreaming (a jar or class dir), otherwise StreamUtil.findInClasspath(StreamJob.class.getName()) locates the containing jar. If both fail it throws IOException 'runtime classes not found: org.apache.hadoop.streaming' before job submission completes.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java:687

  /** @return path to the created Jar file or null if no files are necessary.
   */
  protected String packageJobJar() throws IOException {
    ArrayList<String> unjarFiles = new ArrayList<String>();

    // Runtime code: ship same version of code as self (job submitter code)
    // usually found in: build/contrib or build/hadoop-<version>-dev-streaming.jar

    // First try an explicit spec: it's too hard to find our own location in this case:
    // $HADOOP_HOME/bin/hadoop jar /not/first/on/classpath/custom-hadoop-streaming.jar
    // where findInClasspath() would find the version of hadoop-streaming.jar in $HADOOP_HOME
    String runtimeClasses = config_.get("stream.shipped.hadoopstreaming"); // jar or class dir

    if (runtimeClasses == null) {
      runtimeClasses = StreamUtil.findInClasspath(StreamJob.class.getName());
    }
    if (runtimeClasses == null) {
      throw new IOException("runtime classes not found: " + getClass().getPackage());
    } else {
      msg("Found runtime classes in: " + runtimeClasses);
    }
    if (isLocalHadoop()) {
      // don't package class files (they might get unpackaged in "." and then
      //  hide the intended CLASSPATH entry)
      // we still package everything else (so that scripts and executable are found in
      //  Task workdir like distributed Hadoop)
    } else {
      if (new File(runtimeClasses).isDirectory()) {
        packageFiles_.add(runtimeClasses);
      } else {
        unjarFiles.add(runtimeClasses);
      }
    }
    if (packageFiles_.size() + unjarFiles.size() == 0) {
      return null;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Submit via the canonical form: hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar ...
2. Set the location explicitly: -D stream.shipped.hadoopstreaming=/path/to/hadoop-streaming.jar (or hdfs: URL) so the lookup is bypassed
3. If embedding streaming in an app, ensure StreamJob's classes come from a real jar on the classpath, not exploded classes
4. Verify the streaming jar version matches the cluster's Hadoop version to avoid secondary failures

Example fix

# before (embedded/unresolvable classpath)
java -cp app.jar org.apache.hadoop.streaming.StreamJob -input in -output out -mapper cat
# after (explicit runtime classes)
hadoop jar hadoop-streaming.jar org.apache.hadoop.streaming.StreamJob \
  -D stream.shipped.hadoopstreaming=hdfs://nn/tmp/hadoop-streaming.jar \
  -input in -output out -mapper cat

// before StreamJob: prove the runtime classes are resolvable
String loc = config.get("stream.shipped.hadoopstreaming");
if (loc == null) loc = StreamUtil.findInClasspath(StreamJob.class.getName());
if (loc == null || !new File(loc).exists()) {
  throw new IllegalStateException("set -D stream.shipped.hadoopstreaming=<path to hadoop-streaming jar>");
}

catch IOException from StreamJob/ToolRunner.run; on 'runtime classes not found' fail fast with an actionable message telling the operator to submit via 'hadoop jar hadoop-streaming*.jar' or set stream.shipped.hadoopstreaming.

When it happens

Trigger: Invoking the streaming job in a classloader layout where StreamJob.class is not inside a jar/directory findInClasspath can resolve — e.g. classes loaded from an unpacked WEB-INF/exploded layout, a relocated/shaded jar with odd protection domains, or running through a custom launcher — without setting stream.shipped.hadoopstreaming.

Common situations: Embedding streaming submission inside another application instead of 'hadoop jar hadoop-streaming*.jar', shaded/relocated assemblies that break the classpath-lookup heuristic, or running a stale/trimmed streaming jar; historically also hit when HADOOP_CLASSPATH entries shadow the real streaming jar.

Related errors

• Class not found: {}
• No KeyProviderFactory for ${uri} in ${KEY_PROVIDER_PATH}
• Could not find configured fencing method {}
• Class {} does not implement FenceMethod
• Error creating plugin: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f0c60df3980e6bc5. Report an issue: GitHub.