{"record":{"id":"97fe72ae9c5fb2c1","repo":"apache/seatunnel","slug":"collect-realtime-metrics-failed","errorCode":null,"errorMessage":"Collect realtime metrics failed","messagePattern":"Collect realtime metrics failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/observability/RealtimeMetricsService.java","lineNumber":213,"sourceCode":"        m.put(\"lastCollectEndMs\", lastCollectEndMs);\n        m.put(\"lastRawMetricsFetchCostMs\", lastRawMetricsFetchCostMs);\n        m.put(\"lastRawMetricsBlobs\", lastRawMetricsBlobs);\n        m.put(\"collectFailureCount\", collectFailureCount.get());\n        if (lastCollectError != null) {\n            m.put(\"lastCollectError\", lastCollectError);\n        }\n        return m;\n    }\n\n    private void collectSafely() {\n        lastCollectStartMs = System.currentTimeMillis();\n        try {\n            collectOnce();\n            lastCollectError = null;\n        } catch (Throwable t) {\n            collectFailureCount.incrementAndGet();\n            lastCollectError = t.getClass().getSimpleName() + \": \" + t.getMessage();\n            log.warn(\"Collect realtime metrics failed\", t);\n        } finally {\n            lastCollectEndMs = System.currentTimeMillis();\n        }\n    }\n\n    private void collectOnce() {\n        Set<Long> runningJobIds = getRunningJobIds();\n\n        jobStores.keySet().removeIf(jobId -> !runningJobIds.contains(jobId));\n        jobMetas.keySet().removeIf(jobId -> !runningJobIds.contains(jobId));\n\n        long nowMs = System.currentTimeMillis();\n        Set<Long> enabledJobIds = new HashSet<>();\n        for (Long jobId : runningJobIds) {\n            ObservabilityConfig config = resolveObservabilityConfig(jobId);\n            jobMetas.put(\n                    jobId,\n                    new JobMeta(","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/observability/RealtimeMetricsService.java#L195-L231","documentation":"RealtimeMetricsService runs a periodic collector that fetches metrics via collectOnce(). Any Throwable thrown during a collection cycle is caught, counted, recorded in lastCollectError, and logged at WARN with this message; the service keeps running and retries on the next cycle. It signals that one metrics-collection pass failed, not that the service is dead.","triggerScenarios":"collectOnce() throws — e.g. an underlying IMap/cluster operation fails, a metrics provider throws an unchecked exception, or a serialization error occurs while gathering worker metrics. Any runtime exception in the collection path triggers the log.","commonSituations":"Transient Hazelcast partition/cluster instability during rebalancing; a metrics source throwing on a partially initialized worker; OOME or other JVM-level issues during metric aggregation.","solutions":["Read the attached Throwable 't' in the log for the real root cause and fix that underlying issue.","Check cluster health (Hazelcast logs) for partition migrations or member joins/leaves at the failure time.","Inspect lastCollectError and collectFailureCount via observability endpoints to see whether failures are transient or persistent.","If a specific metric provider throws repeatedly, upgrade/patch that provider and report the exception."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check service health before consuming metrics\nif (service.getLastCollectError() != null && service.getCollectFailureCount() > 0) {\n    // treat current metric snapshot as possibly stale\n}","typeGuard":null,"tryCatchPattern":"// consumer-side: tolerate gaps in realtime metrics\ntry { MetricsSnapshot s = service.snapshot(); ... }\ncatch (StaleMetricsException e) { log.warn(\"metrics stale: {}\", e.getMessage()); }","preventionTips":["Monitor collectFailureCount/lastCollectEndMs for repeated failures","Keep cluster stable during metric collection (avoid frequent restarts)","Alert on lastCollectError persistence rather than single occurrences"],"tags":["metrics","observability","zeta-engine"],"backgroundTag":"upstream-api-error","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}