apache/skywalking · error · RuntimeException

Storage-model cascade failed for metric {}; backend drop did

Error message

Storage-model cascade failed for metric {}; backend drop did not complete. Local state preserved for retry.

What it means

During removeMetric, after local prototype teardown begins, MeterSystem cascades the deletion to the storage layer via ModelRegistry.remove (dropping the model/table in BanyanDB or ES). When that backend drop throws and the StorageManipulationOpt flags escalate-to-caller, the exception is rethrown as this RuntimeException so the caller (Reconciler unregisterBundle) keeps appliedMal[key] and local state intact, letting the next reconcile tick re-enter and retry the cascade.

Source

Thrown at oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java:455

        final Class<?> prototypeClass = def.getMeterPrototype().getClass();

        // Cascade storage-model removal (Hour / Day / Minute) FIRST. ModelRegistry.remove
        // fires whenRemoving on every listener, so each backend's ModelInstaller.dropTable
        // runs — real delete for BanyanDB, no-op for JDBC / Elasticsearch, skipped outright
        // when the caller is a peer-side (WITHOUT_SCHEMA_CHANGE) apply. If a listener throws,
        // ModelRegistry.remove keeps the model in its registry so this retry path stays
        // open: the caller (Reconciler unregisterBundle) preserves appliedMal[key] and the
        // next tick (or operator retry) re-enters this method, finds meterPrototypes still
        // populated, and re-fires the cascade.
        try {
            final ModelRegistry modelCreator = manager.find(CoreModule.NAME)
                                                     .provider()
                                                     .getService(ModelRegistry.class);
            modelCreator.remove(prototypeClass, opt);
        } catch (final Throwable t) {
            log.error("Failed to cascade storage-model removal for metric {}", metricsName, t);
            if (opt.getFlags().isEscalateToCaller()) {
                throw new RuntimeException(
                    "Storage-model cascade failed for metric " + metricsName
                        + "; backend drop did not complete. Local state preserved for retry.",
                    t);
            }
            // Non-escalating opt (peer-side withoutSchemaChange, etc.) — backend drop is
            // suppressed by policy anyway, so a listener throw here is the listener's
            // local bookkeeping misbehaving, not real backend debt. Fall through to
            // clear local state.
        }

        // Backend cascade succeeded (or local-cache-only, where it doesn't matter). Drop
        // the prototype and drain the workers. Worker drain failure is non-fatal and
        // logged: stale workers self-drain within one tick, and the prototype + Model are
        // already gone so future samples can't reach them.
        meterPrototypes.remove(metricsName);
        try {
            MetricsStreamProcessor.getInstance().removeMetric(manager, (Class) prototypeClass);
        } catch (final Throwable t) {

View on GitHub (pinned to 102af09b4a)

Solutions

Check and restore storage backend health (BanyanDB/ES reachable, no auth failures in OAP logs), then let the next reconcile tick retry — local state is preserved by design
If retries keep failing, inspect the nested cause in the log ('Failed to cascade storage-model removal for metric ...') and fix that specific backend error (disk, permissions, schema lock)
As a last resort restart OAP after the backend is healthy; the pending removal re-fires from the applied-rules state

Defensive patterns

Strategy: retry

Validate before calling

// before triggering a rule hot-remove, probe storage health
HealthCheckService hs = manager.find(CoreModule.NAME).provider().getService(HealthCheckService.class);
if (!hs.isHealth()) {
    deferUnregisterToNextTick(); // avoid a doomed cascade
}

Try / catch

try {
    meterSystem.removeMetric(name, StorageManipulationOpt.withSchemaChange());
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().contains("Storage-model cascade failed")) {
        // local state preserved by design — retry on next reconcile tick
        scheduleRetry(name);
    } else {
        throw e;
    }
}

Prevention

Do hot-removals during stable storage windows, not mid-restart of BanyanDB/ES
Monitor the nested cause logged as 'Failed to cascade storage-model removal' and fix backend errors promptly so retries succeed
Keep applied-rule state intact after failure; the retry path depends on it (ModelRegistry keeps the model registered on purpose)

When it happens

Trigger: BanyanDB or Elasticsearch is unreachable or times out while a MAL/LAL rule is being hot-removed; the storage plugin rejects the model drop (permission denied, schema lock, index still open); a metrics listener throws during model removal while escalation is enabled.

Common situations: Hot-removing a meter rule during a storage outage or rolling restart of the storage cluster; removing many metrics at once so the backend throttles some drop requests; version skew between OAP and BanyanDB where the delete-schema RPC changed.

Related errors

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/3a47d996f38f94c8. Report an issue: GitHub.