apache/skywalking · critical · StorageException

local-cache-verify boot: backend resources for model '{model

Error message

local-cache-verify boot: backend resources for model '{modelName}' are not all present — refusing to start. Wait for the init OAP to create them or push the runtime rule. {installInfoMsg}

What it means

ModelInstaller.whenCreating()'s strict boot path with failOnAbsence enabled requires every backend resource backing a model to already exist (created by the init OAP). If the isExists probe reports any missing resource (info.isAllExist() false), the node refuses to start and throws StorageException with the detailed install info — a deliberate fail-fast so a no-init pod never serves traffic against absent tables, instead surfacing the gap as a k8s CrashLoopBackOff.

Source

Thrown at oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java:93

        }

        // Strict verify path — run the read-only existence/shape inspection and surface
        // missing or mismatched resources as fatal so module bootstrap exits (k8s pod
        // backloop). Operator must align with the init OAP first. Distinct from the
        // legacy non-init poll loop further down: that loop waits forever; this path
        // fails fast.
        if (flags.isFailOnAbsence() || flags.isFailOnShapeMismatch()) {
            InstallInfo info = isExists(model, opt);
            if (flags.isFailOnShapeMismatch() && opt.hasShapeMismatch()) {
                final StorageManipulationOpt.ResourceOutcome o = opt.firstShapeMismatch();
                throw new StorageException(
                    "local-cache-verify boot: backend resource '" + (o == null ? model.getName() : o.getResourceName())
                        + "' shape diverges from declared model — refusing to start. "
                        + "Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: "
                        + (o == null ? "n/a" : o.getDiff()));
            }
            if (flags.isFailOnAbsence() && !info.isAllExist()) {
                throw new StorageException(
                    "local-cache-verify boot: backend resources for model '" + model.getName()
                        + "' are not all present — refusing to start. Wait for the init OAP to "
                        + "create them or push the runtime rule. " + info.buildInstallInfoMsg());
            }
            return;
        }

        // Poll loop for the STATIC boot-time path on a non-init OAP: the init OAP owns
        // schema creation, so this node waits until the resource appears rather than
        // creating it. Gated on deferDDLToInitNode (set only on SCHEMA_CREATE_IF_ABSENT),
        // NOT on RunningMode alone — a runtime-rule DSL apply (withSchemaChange) is the
        // operator/main-driven authority and must fall through to createTable below
        // regardless of no-init, because no init OAP knows about a metric created at
        // runtime. Without this, a no-init OAP would block here forever waiting for a
        // resource that only this very apply would ever create.
        if (deferDDLToInitNode(opt)) {
            while (true) {
                boolean allExist;

View on GitHub (pinned to 102af09b4a)

Solutions

Wait for / verify the init OAP has fully booted and created the resources (its logs show successful install), then restart the failing pod.
If the resource is new (runtime rule), push it via the init OAP's /runtime/rule/addOrUpdate so the schema is created centrally.
Check the init OAP's health and storage connectivity — the install message appended to the error names exactly which resources are absent.
As a last resort for local/dev, boot one node without the no-init/fail-fast flags so it performs installation itself.

Defensive patterns

Strategy: retry

Validate before calling

// Operator pre-flight: wait for init OAP readiness + resource existence
// before scaling out no-init replicas (k8s initContainer or Job probing the backend).

Try / catch

// In orchestration (not code): on this StorageException, keep the pod in backoff and retry once the init OAP reports healthy; do not catch-and-continue inside the OAP.

Prevention

Make the init OAP a hard dependency of scale-out: readiness-gate non-init pods on schema presence.
Monitor the init OAP's install logs during first boot of new metrics.
Pre-create schema for new metrics via runtime rule push before deploying the OAP version that queries them.

When it happens

Trigger: Booting a non-init (local-cache-verify / no-init) OAP node before the init OAP has created the model's resources — or when the init OAP failed its own installation, or when the resource was never pushed via the runtime-rule API.

Common situations: Scaling up replicas while the init OAP is still installing schema; init OAP crashed mid-install; storage connectivity from init OAP broken so DDL never landed; a new metric added by config/rule that only the init OAP knows how to create.

Related errors

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/6cc014a2fa54c5ac. Report an issue: GitHub.