apache/hadoop · error · IOException

Parse error

Error message

Parse error

What it means

WNode.parse reads tokens up to the first comma as the InputFormat class name, then requires at least one more token (the path). If the iterator is exhausted right after the comma, i.e. the tbl(...) argument list ends without a path, it throws IOException("Parse error"). An argument list with no comma at all also drains the iterator and lands here.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java:300

      Iterator<Token> i = ll.iterator();
      while (i.hasNext()) {
        Token t = i.next();
        if (TType.COMMA.equals(t.getType())) {
          try {
          	inf = (InputFormat)ReflectionUtils.newInstance(
          			job.getClassByName(sb.toString()),
                job);
          } catch (ClassNotFoundException e) {
            throw (IOException)new IOException().initCause(e);
          } catch (IllegalArgumentException e) {
            throw (IOException)new IOException().initCause(e);
          }
          break;
        }
        sb.append(t.getStr());
      }
      if (!i.hasNext()) {
        throw new IOException("Parse error");
      }
      Token t = i.next();
      if (!TType.QUOT.equals(t.getType())) {
        throw new IOException("Expected quoted string");
      }
      indir = t.getStr();
      // no check for ll.isEmpty() to permit extension
    }

    private JobConf getConf(JobConf job) {
      JobConf conf = new JobConf(job);
      FileInputFormat.setInputPaths(conf, indir);
      conf.setClassLoader(job.getClassLoader());
      return conf;
    }

    public InputSplit[] getSplits(JobConf job, int numSplits)
        throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Always give tbl exactly two arguments: fully-qualified InputFormat class, comma, double-quoted path
2. Generate the expression with CompositeInputFormat.compose(), which always emits both arguments
3. Dry-run setFormat(job) client-side to catch the malformed clause early

Example fix

// before: trailing comma, no path
job.set("mapred.join.expr", "inner(tbl(F,),tbl(F,/b))");

// after
job.set("mapred.join.expr",
    CompositeInputFormat.compose("inner", SequenceFileInputFormat.class, "/a", "/b"));

String expr = CompositeInputFormat.compose("inner",
    SequenceFileInputFormat.class, "/a", "/b"); // always emits class + quoted path
if (expr.contains(",)") || expr.contains("(,")) {
  throw new IllegalArgumentException("empty tbl formal in join expression");
}
job.set("mapred.join.expr", expr);

try {
  new CompositeInputFormat<Object>().setFormat(job);
} catch (IOException e) {
  throw new IllegalArgumentException("malformed tbl(...) clause in mapred.join.expr", e);
}

When it happens

Trigger: tbl(org.apache.hadoop.Foo,) — comma with nothing after it; tbl() with neither class nor path; expressions truncated so the final tbl loses its path argument.

Common situations: String-concatenated expressions that drop the path argument; truncation of long expressions in config XML or shell variables; deleting a path while refactoring.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Expected nodetype
• Unexpected: " + type
• Expected ','
• Expected numtype
• Expected strtype

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dca9308f7cd252f2. Report an issue: GitHub.