apache/dubbo · error · RpcException

Failed to acquire read lock on invokerRefreshLock within tim

Error message

Failed to acquire read lock on invokerRefreshLock within timeout. Timeout: {defaultTimeout}ms, Lock state: [readLockHeld={readLockCount}, writeLockHeld={writeLocked}, writeLockHeldByCurrentThread={writeLockedByCurrentThread}], Service: {serviceKey}

What it means

Thrown by AbstractDirectory.list() when invokerRefreshReadLock.tryLock(DEFAULT_TIMEOUT, MILLISECONDS) returns false — the read lock could not be acquired within the timeout because the write lock is held (or starved) by a long invoker refresh. The message reports the lock state to aid diagnosis. This indicates severe lock contention or a write-side stall during address notification.

Source

Thrown at dubbo-cluster/src/main/java/org/apache/dubbo/rpc/cluster/directory/AbstractDirectory.java:220

    @Override
    public List<Invoker<T>> list(Invocation invocation) throws RpcException {
        if (destroyed) {
            throw new RpcException(
                    "Directory of type " + this.getClass().getSimpleName() + " already destroyed for service "
                            + getConsumerUrl().getServiceKey() + " from registry " + getUrl());
        }

        BitList<Invoker<T>> availableInvokers;
        SingleRouterChain<T> singleChain = null;
        try {
            if (routerChain != null) {
                routerChain.getLock().readLock().lock();
            }
            boolean lockAcquired = false;
            try {
                if (!invokerRefreshReadLock.tryLock(LockUtils.DEFAULT_TIMEOUT, TimeUnit.MILLISECONDS)) {
                    throw new RpcException(
                            "Failed to acquire read lock on invokerRefreshLock within timeout. " + "Timeout: "
                                    + LockUtils.DEFAULT_TIMEOUT + "ms, " + "Lock state: [readLockHeld="
                                    + invokerRefreshLock.getReadLockCount() + ", writeLockHeld="
                                    + invokerRefreshLock.isWriteLocked() + ", writeLockHeldByCurrentThread="
                                    + invokerRefreshLock.isWriteLockedByCurrentThread() + "], Service: "
                                    + getConsumerUrl().getServiceKey());
                }
                lockAcquired = true;
                // use clone to avoid being modified at doList().
                if (invokersInitialized) {
                    availableInvokers = validInvokers.clone();
                } else {
                    availableInvokers = invokers.clone();
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw new RpcException(
                        "Interrupted while acquiring read lock for invoker access, cause: " + e.getMessage(), e);

View on GitHub (pinned to 3a3043227f)

Solutions

Inspect the message's lock state: if writeLockHeld=true for a long time, find the holder (thread dump) and address the slow refresh — reduce provider count, batch notifications, or tune registry.
Increase LockUtils.DEFAULT_TIMEOUT if the environment legitimately needs longer refresh windows (via system property / config), then re-test.
Check for thread-pool starvation on the registry-notify / data-store executors and raise their size or reduce per-notification work.
Upgrade Dubbo — lock handling around refresh has been optimized across versions; ensure you are on a recent patch.

Example fix

// no code fix per se; tune timeout / reduce contention
// e.g. set a longer default lock timeout via system property:
// -Ddubbo.internal.lock.default.timeout=60000
// and capture thread dumps to find the long write-lock holder

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight: avoid calling during heavy refresh if possible
if (directory.isDestroyed()) return null;
// nothing else to validate client-side; contention is server-internal
return directory.list(invocation);

Try / catch

int attempts = 0;
while (true) {
    try {
        return directory.list(invocation);
    } catch (RpcException e) {
        if (++attempts > 2 || !e.getMessage().contains("Failed to acquire read lock")) throw e;
        // back off briefly and retry; contention may clear
        Thread.sleep(50);
    }
}

Prevention

Reduce directory refresh cost (fewer providers, batched notifications).
Monitor writeLockHeld duration; capture thread dumps when this fires to find the holder.
Tune LockUtils.DEFAULT_TIMEOUT upward only if refresh legitimately needs more time.

When it happens

Trigger: A thread holds the invokerRefreshWriteLock for longer than LockUtils.DEFAULT_TIMEOUT while another RPC thread tries to list invokers. Occurs with very large provider address lists, slow refresh logic, a blocked/slow registry notification thread, or thread starvation under heavy load.

Common situations: Thousands of providers causing slow setInvokers(); a slow or hung registry callback holding the write lock; thread pool exhaustion so the writer cannot progress; deadlock between router-chain locks and the directory write lock.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/dubbo@3a3043227f (2026-08-14). Data as JSON: /api/errors/184c8b8cc95b4af0. Report an issue: GitHub.