apache/seatunnel · warning

Ignored malformed cluster health metrics token(s) from membe

Error message

Ignored malformed cluster health metrics token(s) from member {}, tokens={}, rawPrefix={}

What it means

BaseService parses cluster health metrics tokens sent by members (key=value pairs). Tokens that cannot be parsed into a valid key/value are collected and this warning is logged per member, listing the bad tokens and a truncated raw prefix. It is a warn-level diagnostic, not a thrown exception: the malformed tokens are skipped and the rest of the metrics are still applied.

Source

Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/BaseService.java:1464

            if (equalIndex <= 0) {
                if (invalidTokens.size() < INVALID_METRICS_LOG_TOKEN_MAX_COUNT) {
                    invalidTokens.add(truncateForLog(trimmed, INVALID_METRICS_LOG_TOKEN_MAX_LEN));
                }
                continue;
            }
            String key = trimmed.substring(0, equalIndex).trim();
            if (key.isEmpty()) {
                continue;
            }
            String value =
                    equalIndex == trimmed.length() - 1
                            ? ""
                            : trimmed.substring(equalIndex + 1).trim();
            jobInfo.add(key, value);
        }

        if (!invalidTokens.isEmpty() && log.isWarnEnabled() && shouldLogInvalidMetrics()) {
            log.warn(
                    "Ignored malformed cluster health metrics token(s) from member {}, tokens={}, rawPrefix={}",
                    memberAddress == null ? "unknown" : memberAddress,
                    invalidTokens,
                    truncateForLog(input, INVALID_METRICS_LOG_PREFIX_MAX_LEN));
        }

        return jobInfo;
    }

    private static boolean shouldLogInvalidMetrics() {
        long now = System.currentTimeMillis();
        long last = LAST_INVALID_METRICS_LOG_TIME_MS.get();
        if (now - last < INVALID_METRICS_LOG_INTERVAL_MS) {
            return false;
        }
        return LAST_INVALID_METRICS_LOG_TIME_MS.compareAndSet(last, now);
    }

View on GitHub (pinned to cf67b549a7)

Solutions

1. Identify the offending member via the memberAddress in the log and check its SeaTunnel version; upgrade/downgrade so all nodes run a compatible version.
2. Inspect rawPrefix in the log to see the malformed token text and fix whatever emits it (custom agent, wrapper script, extra JVM metrics flags).
3. Verify the member's log/metrics configuration matches the documented format (key=value tokens separated consistently).
4. If the tokens are known-harmless noise, suppress the repeated warns by adjusting logging levels for BaseService.

Example fix

// before (member emits malformed token)
// metrics string: "threadCount=42 garbageToken"
// after: fix the producer so every token is key=value
// metrics string: "threadCount=42 heapUsed=1048576"

// before sending/parsing metrics tokens
// for (String token : metricsString.split(",")) {
//     if (!token.contains("=") || token.split("=", 2)[0].trim().isEmpty()) {
//         log.warn("Skipping malformed metrics token: {}", token);
//         continue;
//     }
// }

boolean isValidMetricsToken(String token) {
    return token != null && token.contains("=") && !token.split("=", 2)[0].trim().isEmpty();
}

When it happens

Trigger: A member publishes a cluster health metrics string containing tokens without the expected '=' separator, or with an empty/unparseable value, e.g. custom or version-mismatched member emitting metrics in an unexpected format.

Common situations: Mixed-version clusters where an older/newer node emits a metrics token format the current node does not understand; custom monitoring agents injecting extra fields; misconfigured jetty/metrics settings adding non key=value tokens.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• GET {} -> HTTP {}
• Logger level request to member {} failed
• Log file name get failed, get log path: {}
• Log file path is empty, no log file path configured in the c
• Log file content is empty, get log path : %s

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/4088a3b28fb89678. Report an issue: GitHub.