{"record":{"id":"3987df2a8f372cf5","repo":"apache/druid","slug":"there-are-too-many-d-pendingnotices","errorCode":null,"errorMessage":"There are too many [%d] pendingNotices.","messagePattern":"There are too many \\[(.+?)\\] pendingNotices\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/query/lookup/LookupReferencesManager.java","lineNumber":298,"sourceCode":"          lookupLoadingSpec.getMode()\n      );\n      return;\n    }\n    addNotice(new LoadNotice(lookupName, lookupExtractorFactoryContainer, lookupConfig.getLookupStartRetries()));\n  }\n\n  public void remove(String lookupName, LookupExtractorFactoryContainer loadedContainer)\n  {\n    Preconditions.checkState(lifecycleLock.awaitStarted(1, TimeUnit.MILLISECONDS));\n    addNotice(new DropNotice(lookupName, loadedContainer));\n  }\n\n  private void addNotice(Notice notice)\n  {\n    atomicallyUpdateStateRef(\n        oldState -> {\n          if (oldState.pendingNotices.size() > 10000) { //don't let pendingNotices grow indefinitely\n            throw new ISE(\"There are too many [%d] pendingNotices.\", oldState.pendingNotices.size());\n          }\n\n          ImmutableList.Builder<Notice> builder = ImmutableList.builder();\n          builder.addAll(oldState.pendingNotices);\n          builder.add(notice);\n\n          return new LookupUpdateState(oldState.lookupMap, builder.build(), oldState.noticesBeingHandled);\n        }\n    );\n    LockSupport.unpark(mainThread);\n  }\n\n  public void submitAsyncLookupTask(Runnable task)\n  {\n    lookupUpdateExecutorService.submit(task);\n  }\n\n  @Override","sourceCodeStart":280,"sourceCodeEnd":316,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/query/lookup/LookupReferencesManager.java#L280-L316","documentation":"addNotice() caps the pending notice queue at 10000 entries to prevent unbounded growth. If more than 10000 notices are already pending (i.e., the main handling thread is not draining them), adding another add/remove notice throws an IllegalStateException.","triggerScenarios":"Calling add() or remove() (directly or via lookup coordinator updates) more than ~10000 times while the background mainThread is stalled, not started, or slower than notice production.","commonSituations":"Bulk lookup sync loops pushing thousands of add/remove calls; main lookup-update thread hung on a slow lookup factory start(); leaked or blocked mainThread after lifecycle problems.","solutions":["Check why the mainThread notice handler is blocked (thread dump; usually stuck in factory start()/close()).","Rate-limit or batch add()/remove() calls instead of issuing them per-key in tight loops.","Verify the manager's lifecycle is started so notices are actually being consumed.","Wait for the queue to drain before issuing more updates (monitor pendingNotices size)."],"exampleFix":"// before: unbounded bulk updates\nfor (Lookup l : allLookups) { manager.add(l.getName(), l.getContainer()); }\n// after: throttle and wait for drain\nwhile (pendingNoticesSize(manager) > 9000) { Thread.sleep(100); }\nfor (Lookup l : allLookups) { manager.add(l.getName(), l.getContainer()); }","handlingStrategy":"validation","validationCode":"// before bulk updates, wait until the notice queue is drained\nwhile (pendingNoticeCount(manager) > 9000) { Thread.sleep(100); }","typeGuard":null,"tryCatchPattern":"try { manager.add(name, container); } catch (IllegalStateException e) { if (e.getMessage().contains(\"pendingNotices\")) { backoffAndRetry(name, container); } else { throw e; } }","preventionTips":["Batch/throttle add() and remove() calls in bulk sync loops","Monitor the size of pendingNotices via logs or metrics","Ensure the manager is started so notices are consumed","Investigate any stall in the main lookup-update thread promptly"],"tags":["backpressure","queue-overflow","lifecycle","threading"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}