{"record":{"id":"bc74322bd5f9934f","repo":"apache/skywalking","slug":"local-cache-verify-boot-backend-resource-resour","errorCode":null,"errorMessage":"local-cache-verify boot: backend resource '{resourceName}' shape diverges from declared model — refusing to start. Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: {diff}","messagePattern":"local-cache-verify boot: backend resource '(.+?)' shape diverges from declared model — refusing to start\\. Reconcile via the init OAP's /runtime/rule/addOrUpdate first\\. diff: (.+?)","errorType":"exception","errorClass":"StorageException","httpStatus":null,"severity":"critical","filePath":"oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java","lineNumber":86,"sourceCode":"                StorageManipulationOpt.Outcome.SKIPPED_NOT_ALLOWED,\n                \"local-cache-only mode; main-node is expected to have installed this resource\");\n            log.debug(\n                \"install: model [{}] not installed; local-cache-only mode — local schema cache refreshed, no isExists probe\",\n                model.getName()\n            );\n            return;\n        }\n\n        // Strict verify path — run the read-only existence/shape inspection and surface\n        // missing or mismatched resources as fatal so module bootstrap exits (k8s pod\n        // backloop). Operator must align with the init OAP first. Distinct from the\n        // legacy non-init poll loop further down: that loop waits forever; this path\n        // fails fast.\n        if (flags.isFailOnAbsence() || flags.isFailOnShapeMismatch()) {\n            InstallInfo info = isExists(model, opt);\n            if (flags.isFailOnShapeMismatch() && opt.hasShapeMismatch()) {\n                final StorageManipulationOpt.ResourceOutcome o = opt.firstShapeMismatch();\n                throw new StorageException(\n                    \"local-cache-verify boot: backend resource '\" + (o == null ? model.getName() : o.getResourceName())\n                        + \"' shape diverges from declared model — refusing to start. \"\n                        + \"Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: \"\n                        + (o == null ? \"n/a\" : o.getDiff()));\n            }\n            if (flags.isFailOnAbsence() && !info.isAllExist()) {\n                throw new StorageException(\n                    \"local-cache-verify boot: backend resources for model '\" + model.getName()\n                        + \"' are not all present — refusing to start. Wait for the init OAP to \"\n                        + \"create them or push the runtime rule. \" + info.buildInstallInfoMsg());\n            }\n            return;\n        }\n\n        // Poll loop for the STATIC boot-time path on a non-init OAP: the init OAP owns\n        // schema creation, so this node waits until the resource appears rather than\n        // creating it. Gated on deferDDLToInitNode (set only on SCHEMA_CREATE_IF_ABSENT),\n        // NOT on RunningMode alone — a runtime-rule DSL apply (withSchemaChange) is the","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/apache/skywalking/blob/102af09b4a56064e22050dded10e2c52e490d040/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java#L68-L104","documentation":"ModelInstaller.whenCreating() has a strict 'local-cache-verify' boot path: when the manipulation flags request fail-on-shape-mismatch, it probes the backend and compares each resource's actual shape against the declared Model. A divergence (columns, types, settings differ — reported via opt.firstShapeMismatch() with a diff) is fatal by design: the pod exits and k8s back-loops rather than run with a schema that would corrupt data. The message points at reconciliation through the init OAP's /runtime/rule/addOrUpdate endpoint.","triggerScenarios":"Booting an OAP node with failOnShapeMismatch enabled (local-cache-verify boot mode) whose storage resource (e.g. a BanyanDB measure/index or ES index) differs in shape from what the current OAP's declared model expects — detected in the isExists probe via opt.hasShapeMismatch().","commonSituations":"Upgrading OAP (or a DSL rule) changed a metric's shape while the backend still holds the old schema; the init OAP created resources with an older rule version; a runtime-rule apply reshaped a metric on some nodes only; rolling upgrades where non-init nodes get the new model before the init OAP reconciles.","solutions":["Trigger the init OAP to reconcile the schema: push the updated rule via /runtime/rule/addOrUpdate (or restart the init OAP so it applies the new declared shape), then let the failing pod restart.","Read the 'diff:' in the message — it names the exact resource and the shape difference (e.g. added/changed column), which tells you which rule or version caused it.","If the divergence is from an abandoned experimental rule, remove/rollback that rule so the declared model matches the backend again.","Do not disable the check in production; instead complete the rolling upgrade so init and non-init nodes agree."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Pre-boot check (operator script): compare declared model shape vs backend\n// via the init OAP's verify/dry-run rule API before starting no-init nodes.","typeGuard":null,"tryCatchPattern":"try { installer.whenCreating(model, opt); } catch (StorageException e) { if (e.getMessage().contains(\"shape diverges\")) { alert(\"schema drift on \" + model.getName() + \": \" + e.getMessage()); haltForOperator(); } throw e; } // never auto-continue on shape drift","preventionTips":["Sequence rolling upgrades: init OAP first (reconciles schema via /runtime/rule/addOrUpdate), then non-init nodes.","Include the rule/schema version in deploy labels so drift is detectable before boot.","Alert on CrashLoopBackOff with this message instead of silencing it — the diff in the message is the remediation map."],"tags":["storage","schema","startup","banyandb","kubernetes","rolling-upgrade"],"backgroundTag":null,"analyzedSha":"102af09b4a56064e22050dded10e2c52e490d040","analyzedAt":"2026-08-14T10:47:52.647Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}