apache/dubbo · critical · IllegalStateException

Too many thread-local indexed variables

Error message

Too many thread-local indexed variables

What it means

Thrown by nextVariableIndex() when the process-global AtomicInteger index for InternalThreadLocal variables exceeds ARRAY_LIST_CAPACITY_MAX_SIZE (Integer.MAX_VALUE - 8) or wraps to negative. Every InternalThreadLocal instance allocates a unique index via this counter; the index never decreases even if the InternalThreadLocal is GC'd. The back-end array per thread cannot grow beyond Integer.MAX_VALUE - 8, so the global cap is enforced. This is a JVM-process-wide limit.

Source

Thrown at dubbo-common/src/main/java/org/apache/dubbo/common/threadlocal/InternalThreadLocalMap.java:99

    public static void remove() {
        Thread thread = Thread.currentThread();
        if (thread instanceof InternalThread) {
            ((InternalThread) thread).setThreadLocalMap(null);
        } else {
            slowThreadLocalMap.remove();
        }
    }

    public static void destroy() {
        slowThreadLocalMap = null;
    }

    public static int nextVariableIndex() {
        int index = NEXT_INDEX.getAndIncrement();
        if (index >= ARRAY_LIST_CAPACITY_MAX_SIZE || index < 0) {
            NEXT_INDEX.set(ARRAY_LIST_CAPACITY_MAX_SIZE);
            throw new IllegalStateException("Too many thread-local indexed variables");
        }
        return index;
    }

    public static int lastVariableIndex() {
        return NEXT_INDEX.get() - 1;
    }

    private InternalThreadLocalMap() {
        indexedVariables = newIndexedVariableTable();
    }

    public Object indexedVariable(int index) {
        Object[] lookup = indexedVariables;
        return index < lookup.length ? lookup[index] : UNSET;
    }

    /**

View on GitHub (pinned to 3a3043227f)

Solutions

1. Find and eliminate the source of unbounded InternalThreadLocal creation — use static final singletons rather than creating new instances per request/connection.
2. If the process has run for a very long time and accumulated legitimate thread-locals, schedule a planned JVM restart before the counter approaches Integer.MAX_VALUE.
3. Search the application and its dependencies for 'new InternalThreadLocal' in non-static contexts and convert them to static final fields.
4. Profile with a heap dump to find the most numerous InternalThreadLocal subclasses and trace their allocation site.

Example fix

// before — creates a new InternalThreadLocal per object instance
public class RequestHandler {
    private InternalThreadLocal<Context> ctx = new InternalThreadLocal<>();
}

// after — single static instance shared across all handlers
public class RequestHandler {
    private static final InternalThreadLocal<Context> CTX = new InternalThreadLocal<>();
}

// Check the global index before creating a new InternalThreadLocal
public static boolean canAllocateThreadLocal() {
    return InternalThreadLocalMap.lastVariableIndex() < InternalThreadLocalMap.ARRAY_LIST_CAPACITY_MAX_SIZE - 1;
}

if (canAllocateThreadLocal()) {
    return new InternalThreadLocal<>();
} else {
    throw new IllegalStateException("Thread-local index pool exhausted — restart JVM");
}

// Catch at the allocation site if you must, but the index never resets —
// a restart is the only real recovery.
try {
    return new InternalThreadLocal<>();
} catch (IllegalStateException e) {
    if (e.getMessage().contains("Too many thread-local")) {
        logger.error("InternalThreadLocal index exhausted, JVM restart required", e);
        // fall back to a regular ThreadLocal if possible
        return null; // or use a plain ThreadLocal alternative
    }
    throw e;
}

When it happens

Trigger: Creating an excessive number of InternalThreadLocal instances over the lifetime of the JVM process. Each new InternalThreadLocal (or subclass) calls nextVariableIndex() at static/instance initialization. The counter is monotonic — it never resets even if old InternalThreadLocals are garbage collected. Hitting ~2.1 billion cumulative allocations, or an integer overflow, triggers this.

Common situations: Long-running JVM processes (weeks/months) that dynamically create InternalThreadLocal instances in hot loops; class-loading churn that repeatedly initializes InternalThreadLocal subclasses; memory/classloader leaks that keep creating new thread-local holders; frameworks built on top of Dubbo that allocate thread-locals per-request instead of per-instance.

AI-assisted analysis of apache/dubbo@3a3043227f (2026-08-14). Data as JSON: /api/errors/62ad4119b2363bcb. Report an issue: GitHub.