apache/skywalking · warning · RemoveException

teardown_deferred

teardown_deferred

Error message

runtime-rule MAL remove failed for {n} metric(s): {name (Cause), ...}

What it means

MalFileApplier.remove aggregates per-metric meterSystem.removeMetric failures into a RemoveException (code teardown_deferred) whose payload is a LinkedHashMap of metric name → Throwable for every metric that could not be removed. Other metrics in the same teardown may have succeeded; the exception preserves exactly the failed subset so a caller can retry just those names. This is a deferred-teardown signal, not a hard abort of the whole uninstall — the surrounding delete flow typically continues and reports residue.

Source

Thrown at oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java:264

     */
    public void remove(final Set<String> metricNames, final StorageManipulationOpt storageOpt) {
        if (metricNames == null || metricNames.isEmpty()) {
            return;
        }
        Map<String, Throwable> failures = null;
        for (final String name : metricNames) {
            try {
                meterSystem.removeMetric(name, storageOpt);
            } catch (final Throwable t) {
                log.warn("runtime-rule MAL remove: failed to remove metric {}", name, t);
                if (failures == null) {
                    failures = new LinkedHashMap<>();
                }
                failures.put(name, t);
            }
        }
        if (failures != null) {
            throw new RemoveException(failures);
        }
    }

    /** Back-compat overload: full-install policy (server-side drop fires). */
    public void remove(final Set<String> metricNames) {
        remove(metricNames, StorageManipulationOpt.withSchemaChange());
    }

    private Rule parse(final String yamlContent, final String sourceName) throws ApplyException {
        try (StringReader reader = new StringReader(yamlContent)) {
            final Rule rule = new Yaml().loadAs(reader, Rule.class);
            if (rule == null) {
                throw new ApplyException("YAML parsed to null — empty or malformed rule file: "
                    + sourceName, null, Collections.emptySet());
            }
            if (rule.getName() == null || rule.getName().isEmpty()) {
                rule.setName(sourceName);
            }

View on GitHub (pinned to 102af09b4a)

Solutions

1. Retry removal for only the failed names carried in the exception's map (the successes are already gone)
2. Resolve the underlying cause per entry: check storage health, and ensure no other installed rule or alarm still references the metric
3. If the metric was already concurrently removed, make removal idempotent or treat that entry as success on retry
4. After retries fail, list MeterSystem contents to confirm residue and clean up manually via the admin API

catch (MalFileApplier.RemoveException e) { for (final Map.Entry<String, Throwable> en : e.getFailures().entrySet()) { log/enqueue(en.getKey()); } // retry only the failed names, successes are already removed }

When it happens

Trigger: Uninstalling/rolling back a MAL file where removeMetric(name, storageOpt) throws for one or more metrics — e.g. storage drop DDL fails, the metric is still referenced by another component, or the meter was already removed concurrently and the removal path treats that as an error.

Common situations: Storage (BanyanDB/ES) temporarily unavailable during rule deletion, leaving orphaned metrics; Two OAP nodes concurrently uninstalling bundles that share metric names; Metrics still attached to active alarm/analysis pipelines that block removal

Related errors

• MAL register failed for {sourceName} (partial)
• MAL compile failed for {sourceName}
• YAML parse failure for {sourceName}
• {causeMessage} — {rootCauseMessage}
• Load meter analyzer configs failed

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/dc382bf857dd1a07. Report an issue: GitHub.