{"record":{"id":"f0c60df3980e6bc5","repo":"apache/hadoop","slug":"runtime-classes-not-found","errorCode":null,"errorMessage":"runtime classes not found: {}","messagePattern":"runtime classes not found: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java","lineNumber":687,"sourceCode":"\n  /** @return path to the created Jar file or null if no files are necessary.\n   */\n  protected String packageJobJar() throws IOException {\n    ArrayList<String> unjarFiles = new ArrayList<String>();\n\n    // Runtime code: ship same version of code as self (job submitter code)\n    // usually found in: build/contrib or build/hadoop-<version>-dev-streaming.jar\n\n    // First try an explicit spec: it's too hard to find our own location in this case:\n    // $HADOOP_HOME/bin/hadoop jar /not/first/on/classpath/custom-hadoop-streaming.jar\n    // where findInClasspath() would find the version of hadoop-streaming.jar in $HADOOP_HOME\n    String runtimeClasses = config_.get(\"stream.shipped.hadoopstreaming\"); // jar or class dir\n\n    if (runtimeClasses == null) {\n      runtimeClasses = StreamUtil.findInClasspath(StreamJob.class.getName());\n    }\n    if (runtimeClasses == null) {\n      throw new IOException(\"runtime classes not found: \" + getClass().getPackage());\n    } else {\n      msg(\"Found runtime classes in: \" + runtimeClasses);\n    }\n    if (isLocalHadoop()) {\n      // don't package class files (they might get unpackaged in \".\" and then\n      //  hide the intended CLASSPATH entry)\n      // we still package everything else (so that scripts and executable are found in\n      //  Task workdir like distributed Hadoop)\n    } else {\n      if (new File(runtimeClasses).isDirectory()) {\n        packageFiles_.add(runtimeClasses);\n      } else {\n        unjarFiles.add(runtimeClasses);\n      }\n    }\n    if (packageFiles_.size() + unjarFiles.size() == 0) {\n      return null;\n    }","sourceCodeStart":669,"sourceCodeEnd":705,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java#L669-L705","documentation":"StreamJob must ship the job submitter's runtime classes (the hadoop-streaming jar) to the cluster so tasks can load streaming code. It first honors the property stream.shipped.hadoopstreaming (a jar or class dir), otherwise StreamUtil.findInClasspath(StreamJob.class.getName()) locates the containing jar. If both fail it throws IOException 'runtime classes not found: org.apache.hadoop.streaming' before job submission completes.","triggerScenarios":"Invoking the streaming job in a classloader layout where StreamJob.class is not inside a jar/directory findInClasspath can resolve — e.g. classes loaded from an unpacked WEB-INF/exploded layout, a relocated/shaded jar with odd protection domains, or running through a custom launcher — without setting stream.shipped.hadoopstreaming.","commonSituations":"Embedding streaming submission inside another application instead of 'hadoop jar hadoop-streaming*.jar', shaded/relocated assemblies that break the classpath-lookup heuristic, or running a stale/trimmed streaming jar; historically also hit when HADOOP_CLASSPATH entries shadow the real streaming jar.","solutions":["Submit via the canonical form: hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar ...","Set the location explicitly: -D stream.shipped.hadoopstreaming=/path/to/hadoop-streaming.jar (or hdfs: URL) so the lookup is bypassed","If embedding streaming in an app, ensure StreamJob's classes come from a real jar on the classpath, not exploded classes","Verify the streaming jar version matches the cluster's Hadoop version to avoid secondary failures"],"exampleFix":"# before (embedded/unresolvable classpath)\njava -cp app.jar org.apache.hadoop.streaming.StreamJob -input in -output out -mapper cat\n# after (explicit runtime classes)\nhadoop jar hadoop-streaming.jar org.apache.hadoop.streaming.StreamJob \\\n  -D stream.shipped.hadoopstreaming=hdfs://nn/tmp/hadoop-streaming.jar \\\n  -input in -output out -mapper cat","handlingStrategy":"validation","validationCode":"// before StreamJob: prove the runtime classes are resolvable\nString loc = config.get(\"stream.shipped.hadoopstreaming\");\nif (loc == null) loc = StreamUtil.findInClasspath(StreamJob.class.getName());\nif (loc == null || !new File(loc).exists()) {\n  throw new IllegalStateException(\"set -D stream.shipped.hadoopstreaming=<path to hadoop-streaming jar>\");\n}","typeGuard":null,"tryCatchPattern":"catch IOException from StreamJob/ToolRunner.run; on 'runtime classes not found' fail fast with an actionable message telling the operator to submit via 'hadoop jar hadoop-streaming*.jar' or set stream.shipped.hadoopstreaming.","preventionTips":["Submit streaming jobs with 'hadoop jar .../hadoop-streaming-*.jar' from the standard distribution layout","When embedding streaming, set stream.shipped.hadoopstreaming explicitly to a real jar path","Keep the client's streaming jar version aligned with the cluster","Avoid shaded/exploded classpath layouts for the classes that submit the job"],"tags":["hadoop-streaming","classpath","job-submission","jar-packaging","configuration"],"backgroundTag":"class-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}