{"record":{"id":"112d35e180d7a541","repo":"apache/druid","slug":"openlineage-emit-queue-full-discarded-d-event","errorCode":null,"errorMessage":"OpenLineage emit queue full, discarded [%,d] events total","messagePattern":"OpenLineage emit queue full, discarded \\[%,d\\] events total","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java","lineNumber":570,"sourceCode":"  /**\n   * Rejection handler that discards events when the emit queue is full, but logs a warning\n   * on the first drop and every {@link #DISCARD_WARNING_INTERVAL} drops thereafter.\n   */\n  private static class DiscardWithWarningPolicy implements RejectedExecutionHandler\n  {\n    private final AtomicLong discardedCount;\n\n    DiscardWithWarningPolicy(AtomicLong discardedCount)\n    {\n      this.discardedCount = discardedCount;\n    }\n\n    @Override\n    public void rejectedExecution(Runnable r, ThreadPoolExecutor executor)\n    {\n      long count = discardedCount.incrementAndGet();\n      if (count == 1 || count % DISCARD_WARNING_INTERVAL == 0) {\n        log.warn(\"OpenLineage emit queue full, discarded [%,d] events total\", count);\n      }\n    }\n  }\n\n}\n","sourceCodeStart":552,"sourceCodeEnd":576,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/openlineage-emitter/src/main/java/org/apache/druid/extensions/openlineage/OpenLineageRequestLogger.java#L552-L576","documentation":"OpenLineageRequestLogger uses a bounded worker queue; when the queue is full, the rejected execution handler discards the event, increments discardedCount, and logs this warning on the first discard and then every DISCARD_WARNING_INTERVAL discards. Events are lost, not queued.","triggerScenarios":"emit() is called faster than the HTTP POST worker can drain the queue — e.g. burst of many SQL queries while the OpenLineage endpoint is slow or unreachable, or thread pool size/queue capacity too small for the event rate.","commonSituations":"Slow OpenLineage collector causing POSTs to block; a query storm (heavy BI refresh) generating lineage events faster than they can be sent; under-provisioned executor after tuning changes.","solutions":["Fix upstream slowness: check why POSTs to the OpenLineage endpoint are slow (see dropped-event warnings)","Increase the emit thread pool size / queue capacity for the OpenLineage request logger","Reduce event volume (e.g. only emit lineage for certain datasources) or batch events","Monitor the discardedCount metric and alert when it starts incrementing"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":["Size the emit queue/thread pool above peak query rate","Fix slow collector endpoints promptly","Reduce lineage event volume (filter datasources)","Alert on discardedCount increases"],"tags":["openlineage","backpressure","queue-overflow","event-loss"],"backgroundTag":"backpressure-queue-overflow","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}