{"record":{"id":"62ad4119b2363bcb","repo":"apache/dubbo","slug":"too-many-thread-local-indexed-variables","errorCode":null,"errorMessage":"Too many thread-local indexed variables","messagePattern":"Too many thread-local indexed variables","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"dubbo-common/src/main/java/org/apache/dubbo/common/threadlocal/InternalThreadLocalMap.java","lineNumber":99,"sourceCode":"\n    public static void remove() {\n        Thread thread = Thread.currentThread();\n        if (thread instanceof InternalThread) {\n            ((InternalThread) thread).setThreadLocalMap(null);\n        } else {\n            slowThreadLocalMap.remove();\n        }\n    }\n\n    public static void destroy() {\n        slowThreadLocalMap = null;\n    }\n\n    public static int nextVariableIndex() {\n        int index = NEXT_INDEX.getAndIncrement();\n        if (index >= ARRAY_LIST_CAPACITY_MAX_SIZE || index < 0) {\n            NEXT_INDEX.set(ARRAY_LIST_CAPACITY_MAX_SIZE);\n            throw new IllegalStateException(\"Too many thread-local indexed variables\");\n        }\n        return index;\n    }\n\n    public static int lastVariableIndex() {\n        return NEXT_INDEX.get() - 1;\n    }\n\n    private InternalThreadLocalMap() {\n        indexedVariables = newIndexedVariableTable();\n    }\n\n    public Object indexedVariable(int index) {\n        Object[] lookup = indexedVariables;\n        return index < lookup.length ? lookup[index] : UNSET;\n    }\n\n    /**","sourceCodeStart":81,"sourceCodeEnd":117,"githubUrl":"https://github.com/apache/dubbo/blob/3a3043227f5571d25eb2889de5bca22f2914843b/dubbo-common/src/main/java/org/apache/dubbo/common/threadlocal/InternalThreadLocalMap.java#L81-L117","documentation":"Thrown by nextVariableIndex() when the process-global AtomicInteger index for InternalThreadLocal variables exceeds ARRAY_LIST_CAPACITY_MAX_SIZE (Integer.MAX_VALUE - 8) or wraps to negative. Every InternalThreadLocal instance allocates a unique index via this counter; the index never decreases even if the InternalThreadLocal is GC'd. The back-end array per thread cannot grow beyond Integer.MAX_VALUE - 8, so the global cap is enforced. This is a JVM-process-wide limit.","triggerScenarios":"Creating an excessive number of InternalThreadLocal instances over the lifetime of the JVM process. Each new InternalThreadLocal (or subclass) calls nextVariableIndex() at static/instance initialization. The counter is monotonic — it never resets even if old InternalThreadLocals are garbage collected. Hitting ~2.1 billion cumulative allocations, or an integer overflow, triggers this.","commonSituations":"Long-running JVM processes (weeks/months) that dynamically create InternalThreadLocal instances in hot loops; class-loading churn that repeatedly initializes InternalThreadLocal subclasses; memory/classloader leaks that keep creating new thread-local holders; frameworks built on top of Dubbo that allocate thread-locals per-request instead of per-instance.","solutions":["Find and eliminate the source of unbounded InternalThreadLocal creation — use static final singletons rather than creating new instances per request/connection.","If the process has run for a very long time and accumulated legitimate thread-locals, schedule a planned JVM restart before the counter approaches Integer.MAX_VALUE.","Search the application and its dependencies for 'new InternalThreadLocal' in non-static contexts and convert them to static final fields.","Profile with a heap dump to find the most numerous InternalThreadLocal subclasses and trace their allocation site."],"exampleFix":"// before — creates a new InternalThreadLocal per object instance\npublic class RequestHandler {\n    private InternalThreadLocal<Context> ctx = new InternalThreadLocal<>();\n}\n\n// after — single static instance shared across all handlers\npublic class RequestHandler {\n    private static final InternalThreadLocal<Context> CTX = new InternalThreadLocal<>();\n}","handlingStrategy":"validation","validationCode":"// Check the global index before creating a new InternalThreadLocal\npublic static boolean canAllocateThreadLocal() {\n    return InternalThreadLocalMap.lastVariableIndex() < InternalThreadLocalMap.ARRAY_LIST_CAPACITY_MAX_SIZE - 1;\n}\n\nif (canAllocateThreadLocal()) {\n    return new InternalThreadLocal<>();\n} else {\n    throw new IllegalStateException(\"Thread-local index pool exhausted — restart JVM\");\n}","typeGuard":null,"tryCatchPattern":"// Catch at the allocation site if you must, but the index never resets —\n// a restart is the only real recovery.\ntry {\n    return new InternalThreadLocal<>();\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"Too many thread-local\")) {\n        logger.error(\"InternalThreadLocal index exhausted, JVM restart required\", e);\n        // fall back to a regular ThreadLocal if possible\n        return null; // or use a plain ThreadLocal alternative\n    }\n    throw e;\n}","preventionTips":["Always declare InternalThreadLocal fields as 'static final' — never per-instance.","Avoid creating InternalThreadLocal in hot loops or per-request.","Monitor InternalThreadLocalMap.lastVariableIndex() in long-running JVMs as a canary.","Schedule periodic JVM restarts for very long-lived processes to reset the monotonic counter."],"tags":["thread-local","memory-leak","jvm-limits","long-running"],"backgroundTag":null,"analyzedSha":"3a3043227f5571d25eb2889de5bca22f2914843b","analyzedAt":"2026-08-14T00:43:19.853Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}