apache/skywalking · error · IllegalExpressionException

The metrics in expression: {expression} must have the same s

Error message

The metrics in expression: {expression} must have the same scope level, but got: {scopeSet}.

What it means

Thrown when the metrics referenced by an alarm rule expression belong to different scope levels (e.g. one ServiceLevel metric and one EndpointLevel metric). An alarm rule is evaluated per entity of one scope, so all includeMetrics must share a single scope; the set of distinct scope names found in ValueColumnMetadata is printed in the message.

Source

Thrown at oap-server/server-alarm-plugin/src/main/java/org/apache/skywalking/oap/server/core/alarm/provider/AlarmRule.java:117

                    "Expression: " + expression + " is not a SINGLE_VALUE result expression.");
            }

            verifyIncludeMetrics(visitor.getIncludeMetrics(), expression);
            this.expression = expression;
            this.includeMetrics = visitor.getIncludeMetrics();
            this.maxTrendRange = visitor.getMaxTrendRange();
        } finally {
            TRACE_CONTEXT.remove();
        }
    }

    private void verifyIncludeMetrics(Set<String> includeMetrics, String expression) throws IllegalExpressionException {
        Set<String> scopeSet = new HashSet<>();
        for (String metricName : includeMetrics) {
            scopeSet.add(ValueColumnMetadata.INSTANCE.getScope(metricName).name());
        }
        if (scopeSet.size() != 1) {
            throw new IllegalExpressionException(
                "The metrics in expression: " + expression + " must have the same scope level, but got: " + scopeSet + ".");
        }
    }
}

View on GitHub (pinned to 102af09b4a)

Solutions

Read the printed scopeSet in the message to see which scopes conflict, then replace the odd metric with the equivalent metric of the dominant scope (e.g. use service_sla instead of endpoint_sla).
If the cross-scope comparison is genuinely needed, express it as two separate rules or pre-aggregate the metric at the needed scope in OAL.
Confirm each metric name in the expression exists and its scope by checking the metrics metadata (MetricsDAO/ValueColumnMetadata, or the UI metric list) before writing the rule.
Fix alarm-settings.yml and restart OAP / re-push the dynamic config; the whole rule file load aborts on this exception.

Example fix

# before
rules:
  mixed_rule:
    expression: service_resp_time > endpoint_percentile(95)  # Service vs Endpoint scopes
# after
rules:
  mixed_rule:
    expression: service_resp_time > service_percentile(95)  # same scope

Defensive patterns

Strategy: validation

Validate before calling

// Before setExpression, assert all referenced metrics share one scope:
Set<String> scopes = includeMetrics.stream()
    .map(m -> ValueColumnMetadata.INSTANCE.getScope(m).name())
    .collect(Collectors.toSet());
if (scopes.size() != 1) throw new IllegalArgumentException("mixed scopes: " + scopes);

Try / catch

catch (IllegalExpressionException e) {
    // message already contains the conflicting scopeSet; log and reject the rule
    log.error("Rejecting alarm rule {}: {}", ruleName, e.getMessage());
}

Prevention

Check each metric's scope in the metrics metadata (UI or ValueColumnMetadata) before combining metrics in one expression.
Keep expressions within one domain (all service_*, or all endpoint_*, etc.).
Add a CI test that loads alarm-settings.yml and asserts every rule constructs successfully.

When it happens

Trigger: Calling AlarmRule.setExpression with an expression mixing metrics of different scopes, e.g. `service_resp_time / endpoint_cpm > 10`, or a metric name that resolves to a different scope than the others in the same expression. The verifyIncludeMetrics loop collects ValueColumnMetadata.INSTANCE.getScope(metricName).name() for every referenced metric and throws when the set size != 1.

Common situations: Building a ratio expression from two metrics that look related but are defined at different levels (service vs endpoint, instance vs service); activating a metric via OAL at a custom scope and reusing it in an existing alarm expression; renaming/moving a metric with a new scope in a new OAP version.

Related errors

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/0e3d064cb4fca2aa. Report an issue: GitHub.