apache/hadoop · error · IllegalArgumentException

Can't have recursive MultithreadedMapper instances.

Error message

Can't have recursive MultithreadedMapper instances.

What it means

MultithreadedMapper.setMapperClass (MultithreadedMapper.java:114-121) validates that the user mapper class is not itself a MultithreadedMapper (or subclass) and otherwise throws IllegalArgumentException. Nesting would spawn thread pools per thread with unbounded recursion, so the check fails fast.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.java:120

  Class<Mapper<K1,V1,K2,V2>> getMapperClass(JobContext job) {
    return (Class<Mapper<K1,V1,K2,V2>>) 
      job.getConfiguration().getClass(MAP_CLASS, Mapper.class);
  }
  
  /**
   * Set the application's mapper class.
   * @param <K1> the map input key type
   * @param <V1> the map input value type
   * @param <K2> the map output key type
   * @param <V2> the map output value type
   * @param job the job to modify
   * @param cls the class to use as the mapper
   */
  public static <K1,V1,K2,V2> 
  void setMapperClass(Job job, 
                      Class<? extends Mapper<K1,V1,K2,V2>> cls) {
    if (MultithreadedMapper.class.isAssignableFrom(cls)) {
      throw new IllegalArgumentException("Can't have recursive " + 
                                         "MultithreadedMapper instances.");
    }
    job.getConfiguration().setClass(MAP_CLASS, cls, Mapper.class);
  }

  /**
   * Run the application's maps using a thread pool.
   */
  @Override
  public void run(Context context) throws IOException, InterruptedException {
    outer = context;
    int numberOfThreads = getNumberOfThreads(context);
    mapClass = getMapperClass(context);
    if (LOG.isDebugEnabled()) {
      LOG.debug("Configuring multithread runner to use " + numberOfThreads + 
                " threads");
    }
    

View on GitHub (pinned to 2add963021)

Solutions

1. Set the runner once: job.setMapperClass(MultithreadedMapper.class), then set the application mapper via MultithreadedMapper.setMapperClass(job, MyBusinessMapper.class)
2. If you subclassed MultithreadedMapper for customization, register the subclass with job.setMapperClass (the runner slot), not with MultithreadedMapper.setMapperClass
3. For custom threading behavior prefer mapreduce.mapper.multithreadedmapper.* settings (mapreduce.map.multithreadedmapper.threads / runners) over subclassing
4. Check the exception at job setup time — it is thrown client-side during configuration, so fix is always in driver code

Example fix

// before
job.setMapperClass(MultithreadedMapper.class);
MultithreadedMapper.setMapperClass(job, MyPoolingMapper.class); // MyPoolingMapper extends MultithreadedMapper -> throws

// after
public class MyPoolingMapper extends MultithreadedMapper<Text, Text, Text, Text> {
  // customization via overrides of hooks (optional)
}
job.setMapperClass(MyPoolingMapper.class);                        // runner in the normal slot
MultithreadedMapper.setMapperClass(job, BusinessMapper.class);    // plain Mapper, no nesting

static void configureMultithreadedMapper(Job job, Class<? extends Mapper<?,?,?,?>> userMapper) {
  if (MultithreadedMapper.class.isAssignableFrom(userMapper))
    throw new IllegalArgumentException("Application mapper must not extend MultithreadedMapper: " + userMapper.getName());
  job.setMapperClass(MultithreadedMapper.class);
  MultithreadedMapper.setMapperClass(job, userMapper);
}

static boolean isRecursiveMapper(Class<?> cls) { return MultithreadedMapper.class.isAssignableFrom(cls); }

try { MultithreadedMapper.setMapperClass(job, cls); } catch (IllegalArgumentException e) { if (e.getMessage().contains("recursive")) throw new IllegalArgumentException("Passed a MultithreadedMapper subclass as the inner mapper — put it in job.setMapperClass instead", e); throw e; }

When it happens

Trigger: Calling MultithreadedMapper.setMapperClass(job, cls) where MultithreadedMapper.class.isAssignableFrom(cls) — i.e. cls is MultithreadedMapper itself or any subclass, including a user class extending MultithreadedMapper to override hooks; also setMapperClass(MultithreadedMapper.class) by mistake instead of the application mapper.

Common situations: Copy-paste from a mapper that extended MultithreadedMapper (old pattern) into new API code; intending to customize threading by subclassing MultithreadedMapper and then registering the subclass as the inner mapper; confusion between job.setMapperClass(MultithreadedMapper.class) (correct, the runner) and the inner setMapperClass (must be the business mapper).

Related errors

• Invalid specification for distributed-cache artifacts of typ
• Unable to parse '{}' as a URI, check the setting for mapredu
• Could not locate MapReduce framework name '{}' in mapreduce.
• MapReduce JobHistory WebApp Address does not contain a valid
• Type mismatch in key from map: expected {keyClassName}, rece

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/46de52618fdeb8e9. Report an issue: GitHub.