apache/hadoop · error · IndexOutOfBoundsException

adding an additional class would exceed the maximum number a

Error message

adding an additional class would exceed the maximum number allowed

What it means

Dynamically registered classes (those not pre-assigned a fixed id) get consecutive positive byte ids via ++newClasses. Once the count would exceed Byte.MAX_VALUE (127), addToMap(clazz) throws IndexOutOfBoundsException - the wire format only has one byte for the id, so a MapWritable cannot carry more than 127 distinct dynamic types.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/AbstractMapWritable.java:96

      if (!c.equals(clazz)) {
        throw new IllegalArgumentException("Id " + id + " exists but maps to " +
            c.getName() + " and not " + clazz.getName());
      }
    }
    classToIdMap.put(clazz, id);
    idToClassMap.put(id, clazz);
  }
  
  /**
   * Add a Class to the maps if it is not already present.
   * @param clazz clazz.
   */
  protected synchronized void addToMap(Class<?> clazz) {
    if (classToIdMap.containsKey(clazz)) {
      return;
    }
    if (newClasses + 1 > Byte.MAX_VALUE) {
      throw new IndexOutOfBoundsException("adding an additional class would" +
      " exceed the maximum number allowed");
    }
    byte id = ++newClasses;
    addToMap(clazz, id);
  }

  /**
   * the Class class for the specified id.
   * @param id id.
   * @return the Class class for the specified id.
   */
  protected Class<?> getClass(byte id) {
    return idToClassMap.get(id);
  }

  /**
   * get id.
   * @return the id for the specified Class.

View on GitHub (pinned to 2add963021)

Solutions

1. Reduce distinct value types: encode values as Text/BytesWritable instead of one Writable class per type
2. Pre-register your types with fixed negative ids (outside -127..-113) if you control the subclass, since predefined ids do not count against the 127
3. Switch containers: ObjectWritable, a custom Writable with an explicit type tag, or a serialization framework without the byte-id cap

Example fix

// before: each domain type eats one dynamic id -> cap at 127
map.put(new Text("user"), new UserWritable(...));
map.put(new Text("order"), new OrderWritable(...));

// after: normalize values to Text, no per-type registration
map.put(new Text("user"), new Text(jsonUser));
map.put(new Text("order"), new Text(jsonOrder));

Set<Class<?>> distinct = map.keySet().stream()
    .map(Object::getClass).collect(Collectors.toSet());
// plus value types actually stored; if distinct types approach 127, normalize instead:
if (distinct.size() > 100) {
  throw new IllegalStateException("MapWritable nearing 127-class cap; encode values as Text");
}

try {
  map.put(key, newValue);
} catch (IndexOutOfBoundsException e) {
  if (e.getMessage().contains("maximum number allowed")) {
    // no more dynamic ids: serialize values as Text/BytesWritable instead
  }
}

When it happens

Trigger: Using MapWritable/SortedMapWritable as a heterogeneous bag and putting values of more than 127 distinct Writable classes into one instance (each new class consumes one id); long-lived accumulators that keep adding new value types.

Common situations: Generic ETL code that boxes many domain types into a MapWritable; migrating maps of many types into Hadoop serialization without pre-registering.

Related errors

• Class {} already registered but maps to {} and not {}
• Id {} exists but maps to {} and not {}
• map cannot be copied: {}
• source map cannot be null
• Exception while get content summary

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c3a2eaa073a7eed4. Report an issue: GitHub.