{"record":{"id":"184c8b8cc95b4af0","repo":"apache/dubbo","slug":"failed-to-acquire-read-lock-on-invokerrefreshlock","errorCode":null,"errorMessage":"Failed to acquire read lock on invokerRefreshLock within timeout. Timeout: {defaultTimeout}ms, Lock state: [readLockHeld={readLockCount}, writeLockHeld={writeLocked}, writeLockHeldByCurrentThread={writeLockedByCurrentThread}], Service: {serviceKey}","messagePattern":"Failed to acquire read lock on invokerRefreshLock within timeout\\. Timeout: (.+?)ms, Lock state: \\[readLockHeld=(.+?), writeLockHeld=(.+?), writeLockHeldByCurrentThread=(.+?)\\], Service: (.+?)","errorType":"exception","errorClass":"RpcException","httpStatus":null,"severity":"error","filePath":"dubbo-cluster/src/main/java/org/apache/dubbo/rpc/cluster/directory/AbstractDirectory.java","lineNumber":220,"sourceCode":"\n    @Override\n    public List<Invoker<T>> list(Invocation invocation) throws RpcException {\n        if (destroyed) {\n            throw new RpcException(\n                    \"Directory of type \" + this.getClass().getSimpleName() + \" already destroyed for service \"\n                            + getConsumerUrl().getServiceKey() + \" from registry \" + getUrl());\n        }\n\n        BitList<Invoker<T>> availableInvokers;\n        SingleRouterChain<T> singleChain = null;\n        try {\n            if (routerChain != null) {\n                routerChain.getLock().readLock().lock();\n            }\n            boolean lockAcquired = false;\n            try {\n                if (!invokerRefreshReadLock.tryLock(LockUtils.DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS)) {\n                    throw new RpcException(\n                            \"Failed to acquire read lock on invokerRefreshLock within timeout. \" + \"Timeout: \"\n                                    + LockUtils.DEFAULT_TIMEOUT + \"ms, \" + \"Lock state: [readLockHeld=\"\n                                    + invokerRefreshLock.getReadLockCount() + \", writeLockHeld=\"\n                                    + invokerRefreshLock.isWriteLocked() + \", writeLockHeldByCurrentThread=\"\n                                    + invokerRefreshLock.isWriteLockedByCurrentThread() + \"], Service: \"\n                                    + getConsumerUrl().getServiceKey());\n                }\n                lockAcquired = true;\n                // use clone to avoid being modified at doList().\n                if (invokersInitialized) {\n                    availableInvokers = validInvokers.clone();\n                } else {\n                    availableInvokers = invokers.clone();\n                }\n            } catch (InterruptedException e) {\n                Thread.currentThread().interrupt();\n                throw new RpcException(\n                        \"Interrupted while acquiring read lock for invoker access, cause: \" + e.getMessage(), e);","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/apache/dubbo/blob/3a3043227f5571d25eb2889de5bca22f2914843b/dubbo-cluster/src/main/java/org/apache/dubbo/rpc/cluster/directory/AbstractDirectory.java#L202-L238","documentation":"Thrown by AbstractDirectory.list() when invokerRefreshReadLock.tryLock(DEFAULT_TIMEOUT, MILLISECONDS) returns false — the read lock could not be acquired within the timeout because the write lock is held (or starved) by a long invoker refresh. The message reports the lock state to aid diagnosis. This indicates severe lock contention or a write-side stall during address notification.","triggerScenarios":"A thread holds the invokerRefreshWriteLock for longer than LockUtils.DEFAULT_TIMEOUT while another RPC thread tries to list invokers. Occurs with very large provider address lists, slow refresh logic, a blocked/slow registry notification thread, or thread starvation under heavy load.","commonSituations":"Thousands of providers causing slow setInvokers(); a slow or hung registry callback holding the write lock; thread pool exhaustion so the writer cannot progress; deadlock between router-chain locks and the directory write lock.","solutions":["Inspect the message's lock state: if writeLockHeld=true for a long time, find the holder (thread dump) and address the slow refresh — reduce provider count, batch notifications, or tune registry.","Increase LockUtils.DEFAULT_TIMEOUT if the environment legitimately needs longer refresh windows (via system property / config), then re-test.","Check for thread-pool starvation on the registry-notify / data-store executors and raise their size or reduce per-notification work.","Upgrade Dubbo — lock handling around refresh has been optimized across versions; ensure you are on a recent patch."],"exampleFix":"// no code fix per se; tune timeout / reduce contention\n// e.g. set a longer default lock timeout via system property:\n// -Ddubbo.internal.lock.default.timeout=60000\n// and capture thread dumps to find the long write-lock holder","handlingStrategy":"retry","validationCode":"// Pre-flight: avoid calling during heavy refresh if possible\nif (directory.isDestroyed()) return null;\n// nothing else to validate client-side; contention is server-internal\nreturn directory.list(invocation);","typeGuard":null,"tryCatchPattern":"int attempts = 0;\nwhile (true) {\n    try {\n        return directory.list(invocation);\n    } catch (RpcException e) {\n        if (++attempts > 2 || !e.getMessage().contains(\"Failed to acquire read lock\")) throw e;\n        // back off briefly and retry; contention may clear\n        Thread.sleep(50);\n    }\n}","preventionTips":["Reduce directory refresh cost (fewer providers, batched notifications).","Monitor writeLockHeld duration; capture thread dumps when this fires to find the holder.","Tune LockUtils.DEFAULT_TIMEOUT upward only if refresh legitimately needs more time."],"tags":["cluster","directory","concurrency","contention","performance"],"backgroundTag":null,"analyzedSha":"3a3043227f5571d25eb2889de5bca22f2914843b","analyzedAt":"2026-08-14T00:43:19.853Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}