{"record":{"id":"3a47d996f38f94c8","repo":"apache/skywalking","slug":"storage-model-cascade-failed-for-metric-backen","errorCode":null,"errorMessage":"Storage-model cascade failed for metric {}; backend drop did not complete. Local state preserved for retry.","messagePattern":"Storage-model cascade failed for metric (.+?); backend drop did not complete\\. Local state preserved for retry\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java","lineNumber":455,"sourceCode":"        final Class<?> prototypeClass = def.getMeterPrototype().getClass();\n\n        // Cascade storage-model removal (Hour / Day / Minute) FIRST. ModelRegistry.remove\n        // fires whenRemoving on every listener, so each backend's ModelInstaller.dropTable\n        // runs — real delete for BanyanDB, no-op for JDBC / Elasticsearch, skipped outright\n        // when the caller is a peer-side (WITHOUT_SCHEMA_CHANGE) apply. If a listener throws,\n        // ModelRegistry.remove keeps the model in its registry so this retry path stays\n        // open: the caller (Reconciler unregisterBundle) preserves appliedMal[key] and the\n        // next tick (or operator retry) re-enters this method, finds meterPrototypes still\n        // populated, and re-fires the cascade.\n        try {\n            final ModelRegistry modelCreator = manager.find(CoreModule.NAME)\n                                                     .provider()\n                                                     .getService(ModelRegistry.class);\n            modelCreator.remove(prototypeClass, opt);\n        } catch (final Throwable t) {\n            log.error(\"Failed to cascade storage-model removal for metric {}\", metricsName, t);\n            if (opt.getFlags().isEscalateToCaller()) {\n                throw new RuntimeException(\n                    \"Storage-model cascade failed for metric \" + metricsName\n                        + \"; backend drop did not complete. Local state preserved for retry.\",\n                    t);\n            }\n            // Non-escalating opt (peer-side withoutSchemaChange, etc.) — backend drop is\n            // suppressed by policy anyway, so a listener throw here is the listener's\n            // local bookkeeping misbehaving, not real backend debt. Fall through to\n            // clear local state.\n        }\n\n        // Backend cascade succeeded (or local-cache-only, where it doesn't matter). Drop\n        // the prototype and drain the workers. Worker drain failure is non-fatal and\n        // logged: stale workers self-drain within one tick, and the prototype + Model are\n        // already gone so future samples can't reach them.\n        meterPrototypes.remove(metricsName);\n        try {\n            MetricsStreamProcessor.getInstance().removeMetric(manager, (Class) prototypeClass);\n        } catch (final Throwable t) {","sourceCodeStart":437,"sourceCodeEnd":473,"githubUrl":"https://github.com/apache/skywalking/blob/102af09b4a56064e22050dded10e2c52e490d040/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java#L437-L473","documentation":"During removeMetric, after local prototype teardown begins, MeterSystem cascades the deletion to the storage layer via ModelRegistry.remove (dropping the model/table in BanyanDB or ES). When that backend drop throws and the StorageManipulationOpt flags escalate-to-caller, the exception is rethrown as this RuntimeException so the caller (Reconciler unregisterBundle) keeps appliedMal[key] and local state intact, letting the next reconcile tick re-enter and retry the cascade.","triggerScenarios":"BanyanDB or Elasticsearch is unreachable or times out while a MAL/LAL rule is being hot-removed; the storage plugin rejects the model drop (permission denied, schema lock, index still open); a metrics listener throws during model removal while escalation is enabled.","commonSituations":"Hot-removing a meter rule during a storage outage or rolling restart of the storage cluster; removing many metrics at once so the backend throttles some drop requests; version skew between OAP and BanyanDB where the delete-schema RPC changed.","solutions":["Check and restore storage backend health (BanyanDB/ES reachable, no auth failures in OAP logs), then let the next reconcile tick retry — local state is preserved by design","If retries keep failing, inspect the nested cause in the log ('Failed to cascade storage-model removal for metric ...') and fix that specific backend error (disk, permissions, schema lock)","As a last resort restart OAP after the backend is healthy; the pending removal re-fires from the applied-rules state"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before triggering a rule hot-remove, probe storage health\nHealthCheckService hs = manager.find(CoreModule.NAME).provider().getService(HealthCheckService.class);\nif (!hs.isHealth()) {\n    deferUnregisterToNextTick(); // avoid a doomed cascade\n}","typeGuard":null,"tryCatchPattern":"try {\n    meterSystem.removeMetric(name, StorageManipulationOpt.withSchemaChange());\n} catch (RuntimeException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Storage-model cascade failed\")) {\n        // local state preserved by design — retry on next reconcile tick\n        scheduleRetry(name);\n    } else {\n        throw e;\n    }\n}","preventionTips":["Do hot-removals during stable storage windows, not mid-restart of BanyanDB/ES","Monitor the nested cause logged as 'Failed to cascade storage-model removal' and fix backend errors promptly so retries succeed","Keep applied-rule state intact after failure; the retry path depends on it (ModelRegistry keeps the model registered on purpose)"],"tags":["meter-system","storage","banyandb","elasticsearch","hot-remove","retry"],"backgroundTag":null,"analyzedSha":"102af09b4a56064e22050dded10e2c52e490d040","analyzedAt":"2026-08-14T10:47:52.647Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}