apache/druid · warning

Multiple monitors on the same cache causing race conditions…

Error message

Multiple monitors on the same cache causing race conditions and unreliable stats reporting

What it means

CaffeineCache monitors cache statistics by snapshotting Caffeine stats and atomically swapping them via a priorStats AtomicReference with compareAndSet. If two monitors poll the same cache concurrently, the CAS fails — one monitor already replaced the snapshot — making reported delta stats unreliable. The monitor logs a warning (with an ISE for stack trace) and continues emitting possibly skewed metrics.

Solutions

1. Find and remove the duplicate monitor registration so only one monitor references the Caffeine cache.
2. Review Guice bindings/configuration that create or register the cache monitor; ensure a single binding.
3. Restart the process after fixing configuration to clear duplicated monitor state.
4. Treat the emitted caffeine cache metrics as unreliable until the duplicate is removed.

// ensure only one monitor per cache instance at registration time
Set<CaffeineCache> registered = new HashSet<>();
if (!registered.add(cache)) { throw new IllegalStateException("Duplicate monitor for cache"); }

When it happens

Trigger: doMonitor is invoked for a Caffeine cache while another monitor instance is also registered for the same cache — typically the same cache object was registered as a monitor twice (e.g. duplicate injector bindings or registering the cache in two monitor lists).

Common situations: Duplicate cache registration in Druid monitoring configuration; extension or module creating two monitors bound to the same cache instance; hot-reload of monitor configuration without deregistering the old monitor.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Could not populate cache
• Expected 0 resource count, got
• Indices to merge contained metric
• Unable to get SHA1 digest instance
• Unable to write to cache

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8163330a1bdaff2f. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/client/cache/CaffeineCache.java:167

  }

  @Override
  public void doMonitor(ServiceEmitter emitter)
  {
    final CacheStats oldStats = priorStats.get();
    final CacheStats newStats = cache.stats();
    final CacheStats deltaStats = newStats.minus(oldStats);

    final ServiceMetricEvent.Builder builder = ServiceMetricEvent.builder();
    emitter.emit(builder.setMetric("query/cache/caffeine/delta/requests", deltaStats.requestCount()));
    emitter.emit(builder.setMetric("query/cache/caffeine/total/requests", newStats.requestCount()));
    emitter.emit(builder.setMetric("query/cache/caffeine/delta/loadTime", deltaStats.totalLoadTime()));
    emitter.emit(builder.setMetric("query/cache/caffeine/total/loadTime", newStats.totalLoadTime()));
    emitter.emit(builder.setMetric("query/cache/caffeine/delta/evictionBytes", deltaStats.evictionWeight()));
    emitter.emit(builder.setMetric("query/cache/caffeine/total/evictionBytes", newStats.evictionWeight()));
    if (!priorStats.compareAndSet(oldStats, newStats)) {
      // ISE for stack trace
      log.warn(
          new IllegalStateException("Multiple monitors"),
          "Multiple monitors on the same cache causing race conditions and unreliable stats reporting"
      );
    }
  }

  @VisibleForTesting
  Cache<NamedKey, byte[]> getCache()
  {
    return cache;
  }

  private byte[] deserialize(byte[] bytes)
  {
    if (bytes == null) {
      return null;
    }
    final int decompressedLen = ByteBuffer.wrap(bytes).getInt();

View on GitHub (pinned to 9b90983fd2)