apache/hadoop · error · IOException
Parse error
Error message
Parse error
What it means
Thrown by the shift-reduce parser for MapReduce join expressions (org.apache.hadoop.mapreduce.lib.join.Parser). A 'tbl(...)' wrapped-input node must consist of an InputFormat class name, a comma, and a quoted input path. 'Parse error' means the token stream inside tbl() ended before both arguments were found — almost always a missing comma and/or missing path argument after the InputFormat class name.
Source
Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java:315
StringBuilder sb = new StringBuilder();
Iterator<Token> i = ll.iterator();
while (i.hasNext()) {
Token t = i.next();
if (TType.COMMA.equals(t.getType())) {
try {
inf = (InputFormat<?, ?>)ReflectionUtils.newInstance(
conf.getClassByName(sb.toString()), conf);
} catch (ClassNotFoundException e) {
throw new IOException(e);
} catch (IllegalArgumentException e) {
throw new IOException(e);
}
break;
}
sb.append(t.getStr());
}
if (!i.hasNext()) {
throw new IOException("Parse error");
}
Token t = i.next();
if (!TType.QUOT.equals(t.getType())) {
throw new IOException("Expected quoted string");
}
indir = t.getStr();
// no check for ll.isEmpty() to permit extension
}
private Configuration getConf(Configuration jconf) throws IOException {
Job job = Job.getInstance(jconf);
FileInputFormat.setInputPaths(job, indir);
return job.getConfiguration();
}
public List<InputSplit> getSplits(JobContext context)
throws IOException, InterruptedException {
return inf.getSplits(View on GitHub (pinned to 2add963021)
Solutions
- Add the missing ', "<input path>"' second argument to the tbl(...) node, e.g. tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat, "/data/join/a")
- Build the expression with CompositeInputFormat.compose("inner", TextInputFormat.class, "/a", "/b") instead of hand-writing it
- Validate the expression with Parser.parse(expr, conf) (or CompositeInputFormat.setFormat(conf)) in a unit test before submitting the job
- Check every tbl( node in the expression: each must contain exactly class-name, comma, double-quoted path
Example fix
// before
conf.set("mapreduce.join.expr",
"inner(tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat)," +
"tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat, \"/data/b\"))");
// after
conf.set("mapreduce.join.expr",
"inner(tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat, \"/data/a\")," +
"tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat, \"/data/b\"))");
// or simpler
conf.set("mapreduce.join.expr",
CompositeInputFormat.compose("inner", TextInputFormat.class, "/data/a", "/data/b")); Defensive patterns
Strategy: validation
Validate before calling
static void requireValidJoinExpr(String expr) {
// cheap structural checks before job submission
if (expr == null || expr.trim().isEmpty()) throw new IllegalArgumentException("empty join expression");
// every tbl( must be followed by class, comma, quoted path: regex smoke test
java.util.regex.Pattern p = java.util.regex.Pattern.compile(
"tbl\\(\\s*[\\w$.]+\\s*,\\s*\"[^\"]*\"\\s*\\)");
java.util.regex.Matcher m = p.matcher(expr);
int tbls = expr.split("tbl\\(", -1).length - 1;
int ok = 0; while (m.find()) ok++;
if (tbls != ok) throw new IllegalArgumentException("each tbl() needs <class>,\"<path>\"");
} Try / catch
try { Parser.parse(expr, conf); } catch (IOException e) { throw new IllegalArgumentException("Invalid mapreduce.join.expr: " + expr, e); } Prevention
- Build expressions with CompositeInputFormat.compose(...) instead of string literals
- Unit-test Parser.parse(expr, conf) for every hand-written expression
- Keep a lint rule: each tbl( node must match tbl(<class>, "<path>")
When it happens
Trigger: WNode.parse (Parser.java:296-316) consumes tokens until it sees a COMMA, which terminates the InputFormat class name. It then requires at least one more token (the quoted path). The error fires when i.hasNext() is false, e.g. the expression 'tbl(org.apache.hadoop.mapreduce.lib.input.TextInputFormat)' with no ', "path"' suffix, or a tbl() whose arguments stop right after the class token.
Common situations: Hand-writing the mapreduce.join.expr property or building the expression string manually instead of using CompositeInputFormat.compose(); deleting the path argument while editing; copy-paste that loses the trailing comma+path; shell/XML escaping that truncates the quoted string.
Understand the failure class
- Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.
Related errors
AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22).
Data as JSON: /api/errors/2320c79994b71c9e.
Report an issue: GitHub.