apache/skywalking · critical · StorageException

local-cache-verify boot: backend resource '{resourceName}' s

Error message

local-cache-verify boot: backend resource '{resourceName}' shape diverges from declared model — refusing to start. Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: {diff}

What it means

ModelInstaller.whenCreating() has a strict 'local-cache-verify' boot path: when the manipulation flags request fail-on-shape-mismatch, it probes the backend and compares each resource's actual shape against the declared Model. A divergence (columns, types, settings differ — reported via opt.firstShapeMismatch() with a diff) is fatal by design: the pod exits and k8s back-loops rather than run with a schema that would corrupt data. The message points at reconciliation through the init OAP's /runtime/rule/addOrUpdate endpoint.

Source

Thrown at oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java:86

                StorageManipulationOpt.Outcome.SKIPPED_NOT_ALLOWED,
                "local-cache-only mode; main-node is expected to have installed this resource");
            log.debug(
                "install: model [{}] not installed; local-cache-only mode — local schema cache refreshed, no isExists probe",
                model.getName()
            );
            return;
        }

        // Strict verify path — run the read-only existence/shape inspection and surface
        // missing or mismatched resources as fatal so module bootstrap exits (k8s pod
        // backloop). Operator must align with the init OAP first. Distinct from the
        // legacy non-init poll loop further down: that loop waits forever; this path
        // fails fast.
        if (flags.isFailOnAbsence() || flags.isFailOnShapeMismatch()) {
            InstallInfo info = isExists(model, opt);
            if (flags.isFailOnShapeMismatch() && opt.hasShapeMismatch()) {
                final StorageManipulationOpt.ResourceOutcome o = opt.firstShapeMismatch();
                throw new StorageException(
                    "local-cache-verify boot: backend resource '" + (o == null ? model.getName() : o.getResourceName())
                        + "' shape diverges from declared model — refusing to start. "
                        + "Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: "
                        + (o == null ? "n/a" : o.getDiff()));
            }
            if (flags.isFailOnAbsence() && !info.isAllExist()) {
                throw new StorageException(
                    "local-cache-verify boot: backend resources for model '" + model.getName()
                        + "' are not all present — refusing to start. Wait for the init OAP to "
                        + "create them or push the runtime rule. " + info.buildInstallInfoMsg());
            }
            return;
        }

        // Poll loop for the STATIC boot-time path on a non-init OAP: the init OAP owns
        // schema creation, so this node waits until the resource appears rather than
        // creating it. Gated on deferDDLToInitNode (set only on SCHEMA_CREATE_IF_ABSENT),
        // NOT on RunningMode alone — a runtime-rule DSL apply (withSchemaChange) is the

View on GitHub (pinned to 102af09b4a)

Solutions

1. Trigger the init OAP to reconcile the schema: push the updated rule via /runtime/rule/addOrUpdate (or restart the init OAP so it applies the new declared shape), then let the failing pod restart.
2. Read the 'diff:' in the message — it names the exact resource and the shape difference (e.g. added/changed column), which tells you which rule or version caused it.
3. If the divergence is from an abandoned experimental rule, remove/rollback that rule so the declared model matches the backend again.
4. Do not disable the check in production; instead complete the rolling upgrade so init and non-init nodes agree.

// Pre-boot check (operator script): compare declared model shape vs backend
// via the init OAP's verify/dry-run rule API before starting no-init nodes.

try { installer.whenCreating(model, opt); } catch (StorageException e) { if (e.getMessage().contains("shape diverges")) { alert("schema drift on " + model.getName() + ": " + e.getMessage()); haltForOperator(); } throw e; } // never auto-continue on shape drift

When it happens

Trigger: Booting an OAP node with failOnShapeMismatch enabled (local-cache-verify boot mode) whose storage resource (e.g. a BanyanDB measure/index or ES index) differs in shape from what the current OAP's declared model expects — detected in the isExists probe via opt.hasShapeMismatch().

Common situations: Upgrading OAP (or a DSL rule) changed a metric's shape while the backend still holds the old schema; the init OAP created resources with an older rule version; a runtime-rule apply reshaped a metric on some nodes only; rolling upgrades where non-init nodes get the new model before the init OAP reconciles.

Related errors

• local-cache-verify boot: backend resources for model '{model
• schema fence failed for {sourceName}
• Metric TTL should be at least 2 days, current value is {}
• Record TTL should be at least 2 days, current value is {}
• Storage-model cascade failed for metric {}; backend drop did

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/bc74322bd5f9934f. Report an issue: GitHub.