apache/seatunnel · warning

Collect realtime metrics failed

Error message

Collect realtime metrics failed

What it means

RealtimeMetricsService runs a periodic collector that fetches metrics via collectOnce(). Any Throwable thrown during a collection cycle is caught, counted, recorded in lastCollectError, and logged at WARN with this message; the service keeps running and retries on the next cycle. It signals that one metrics-collection pass failed, not that the service is dead.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/observability/RealtimeMetricsService.java:213

        m.put("lastCollectEndMs", lastCollectEndMs);
        m.put("lastRawMetricsFetchCostMs", lastRawMetricsFetchCostMs);
        m.put("lastRawMetricsBlobs", lastRawMetricsBlobs);
        m.put("collectFailureCount", collectFailureCount.get());
        if (lastCollectError != null) {
            m.put("lastCollectError", lastCollectError);
        }
        return m;
    }

    private void collectSafely() {
        lastCollectStartMs = System.currentTimeMillis();
        try {
            collectOnce();
            lastCollectError = null;
        } catch (Throwable t) {
            collectFailureCount.incrementAndGet();
            lastCollectError = t.getClass().getSimpleName() + ": " + t.getMessage();
            log.warn("Collect realtime metrics failed", t);
        } finally {
            lastCollectEndMs = System.currentTimeMillis();
        }
    }

    private void collectOnce() {
        Set<Long> runningJobIds = getRunningJobIds();

        jobStores.keySet().removeIf(jobId -> !runningJobIds.contains(jobId));
        jobMetas.keySet().removeIf(jobId -> !runningJobIds.contains(jobId));

        long nowMs = System.currentTimeMillis();
        Set<Long> enabledJobIds = new HashSet<>();
        for (Long jobId : runningJobIds) {
            ObservabilityConfig config = resolveObservabilityConfig(jobId);
            jobMetas.put(
                    jobId,
                    new JobMeta(

View on GitHub (pinned to cf67b549a7)

Solutions

1. Read the attached Throwable 't' in the log for the real root cause and fix that underlying issue.
2. Check cluster health (Hazelcast logs) for partition migrations or member joins/leaves at the failure time.
3. Inspect lastCollectError and collectFailureCount via observability endpoints to see whether failures are transient or persistent.
4. If a specific metric provider throws repeatedly, upgrade/patch that provider and report the exception.

// check service health before consuming metrics
if (service.getLastCollectError() != null && service.getCollectFailureCount() > 0) {
    // treat current metric snapshot as possibly stale
}

// consumer-side: tolerate gaps in realtime metrics
try { MetricsSnapshot s = service.snapshot(); ... }
catch (StaleMetricsException e) { log.warn("metrics stale: {}", e.getMessage()); }

When it happens

Trigger: collectOnce() throws — e.g. an underlying IMap/cluster operation fails, a metrics provider throws an unchecked exception, or a serialization error occurs while gathering worker metrics. Any runtime exception in the collection path triggers the log.

Common situations: Transient Hazelcast partition/cluster instability during rebalancing; a metrics source throwing on a partially initialized worker; OOME or other JVM-level issues during metric aggregation.

Related errors

• Metrics snapshots are updated through merge semantics rather
• Ignoring attempted add of a metric due to being null for nam
• Name collision: MetricsContext already contains a Metric wit
• ${CONNECTOR_JAR_HA_STORAGE_TYPE} must in [localfile, hdfs]
• Failed to call factoryIdentifier method.

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/97fe72ae9c5fb2c1. Report an issue: GitHub.