Timed out waiting for the observer-health lock; updating unl

Error message

Timed out waiting for the observer-health lock; updating unlocked

What it means

Observer-health ledger mutations take a file lock (lockPath) via a synchronous retry loop with a deadline. If the deadline passes without acquiring it, the code warns and performs mutate() anyway — the update still happens, with a small risk of interleaving with a competing circuit-breaker writer. The lock is only released in finally when it was actually held.

Source

Thrown at src/shared/observer-health.ts:205

      } catch {
        // Holder released between our failed create and the stat — retry.
        continue;
      }
      if (Date.now() - mtimeMs > LEDGER_LOCK_STALE_MS) {
        try {
          unlinkSync(lockPath);
        } catch {
          // A competing breaker won, or the fs refused the delete; the retry
          // loop re-evaluates either way.
        }
        continue;
      }
      sleepSync(LEDGER_LOCK_RETRY_MS);
    }
  }

  if (!held && Date.now() >= deadline) {
    logger.warn('SESSION', 'Timed out waiting for the observer-health lock; updating unlocked', { lockPath });
  }

  try {
    return mutate();
  } finally {
    if (held) {
      try {
        unlinkSync(lockPath);
      } catch {
        // Already broken as stale by a waiter — nothing to release.
      }
    }
  }
}

export function recordObserverFailure(
  provider: string,
  error: string | ObserverFailureDetail,

View on GitHub (pinned to 8bc631a71a)

Solutions

Treat as usually benign — one unlocked ledger write; re-check observer health status afterwards to confirm consistency.
Reduce the number of simultaneously launched sessions that hammer the same ledger.
If persistent, inspect for a zombie holder: lsof on the lock path.
Restart the worker to clear lock contention.

Defensive patterns

Strategy: fallback

Validate before calling

const held = acquireLedgerLockWithDeadline(lockPath, deadline);
if (!held) {
  // decide explicitly: proceed unlocked (current behavior) or skip
  logger.warn('SESSION', 'Timed out waiting for the observer-health lock; updating unlocked', { lockPath });
}

Try / catch

try {
  return mutate();
} finally {
  if (held) safeUnlink(lockPath);
}

Prevention

Avoid launching many simultaneous hook processes against one observer-health ledger.
Keep the ledger on a local, fast filesystem — not network storage.
Periodically clear dead lock files; the stale-break logic handles it, but fewer waiters means fewer unlocked writes.

When it happens

Trigger: Another process holds the ledger lock past the deadline: many concurrent hook processes mutating the same observer-health ledger, a slow filesystem dragging the retry loop, or GC/paging pauses on the holder.

Common situations: Bursty parallel Claude sessions each running hooks against one ledger; ledger stored on network/slow storage; a crashed holder whose lock got broken as stale, causing retry churn.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of thedotmack/claude-mem@8bc631a71a (2026-08-20). Data as JSON: /api/errors/100765f54cbeafa0. Report an issue: GitHub.