{"record":{"id":"8163330a1bdaff2f","repo":"apache/druid","slug":"multiple-monitors-on-the-same-cache-causing-race-c","errorCode":null,"errorMessage":"Multiple monitors on the same cache causing race conditions and unreliable stats reporting","messagePattern":"Multiple monitors on the same cache causing race conditions and unreliable stats reporting","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"server/src/main/java/org/apache/druid/client/cache/CaffeineCache.java","lineNumber":167,"sourceCode":"  }\n\n  @Override\n  public void doMonitor(ServiceEmitter emitter)\n  {\n    final CacheStats oldStats = priorStats.get();\n    final CacheStats newStats = cache.stats();\n    final CacheStats deltaStats = newStats.minus(oldStats);\n\n    final ServiceMetricEvent.Builder builder = ServiceMetricEvent.builder();\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/delta/requests\", deltaStats.requestCount()));\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/total/requests\", newStats.requestCount()));\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/delta/loadTime\", deltaStats.totalLoadTime()));\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/total/loadTime\", newStats.totalLoadTime()));\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/delta/evictionBytes\", deltaStats.evictionWeight()));\n    emitter.emit(builder.setMetric(\"query/cache/caffeine/total/evictionBytes\", newStats.evictionWeight()));\n    if (!priorStats.compareAndSet(oldStats, newStats)) {\n      // ISE for stack trace\n      log.warn(\n          new IllegalStateException(\"Multiple monitors\"),\n          \"Multiple monitors on the same cache causing race conditions and unreliable stats reporting\"\n      );\n    }\n  }\n\n  @VisibleForTesting\n  Cache<NamedKey, byte[]> getCache()\n  {\n    return cache;\n  }\n\n  private byte[] deserialize(byte[] bytes)\n  {\n    if (bytes == null) {\n      return null;\n    }\n    final int decompressedLen = ByteBuffer.wrap(bytes).getInt();","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/client/cache/CaffeineCache.java#L149-L185","documentation":"CaffeineCache monitors cache statistics by snapshotting Caffeine stats and atomically swapping them via a priorStats AtomicReference with compareAndSet. If two monitors poll the same cache concurrently, the CAS fails — one monitor already replaced the snapshot — making reported delta stats unreliable. The monitor logs a warning (with an ISE for stack trace) and continues emitting possibly skewed metrics.","triggerScenarios":"doMonitor is invoked for a Caffeine cache while another monitor instance is also registered for the same cache — typically the same cache object was registered as a monitor twice (e.g. duplicate injector bindings or registering the cache in two monitor lists).","commonSituations":"Duplicate cache registration in Druid monitoring configuration; extension or module creating two monitors bound to the same cache instance; hot-reload of monitor configuration without deregistering the old monitor.","solutions":["Find and remove the duplicate monitor registration so only one monitor references the Caffeine cache.","Review Guice bindings/configuration that create or register the cache monitor; ensure a single binding.","Restart the process after fixing configuration to clear duplicated monitor state.","Treat the emitted caffeine cache metrics as unreliable until the duplicate is removed."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// ensure only one monitor per cache instance at registration time\nSet<CaffeineCache> registered = new HashSet<>();\nif (!registered.add(cache)) { throw new IllegalStateException(\"Duplicate monitor for cache\"); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Register each cache with exactly one monitor","Review Guice bindings for duplicate cache providers","After config changes, restart to drop stale monitors"],"tags":["druid","cache","monitoring","caffeine","metrics"],"backgroundTag":"conflicting-config-options","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}