apache/skywalking · error · ApplyException

schema fence failed for {sourceName}

Error message

schema fence failed for {sourceName}

What it means

ApplyException thrown when storageOpt.runDeferredFence() raises StorageException after all metric DDL for the file was fired. The deferred schema fence is the single barrier that waits for schema propagation (e.g. BanyanDB mod_revision watermark across data nodes); only a barrier transport error throws — a mere timeout is a non-fatal WARN inside the closure. On throw, layer claims are rolled back and metricNames are passed as the rollback set.

Source

Thrown at oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java:189

            layerRegistry.rollback(appliedClaims);
            throw new ApplyException("MAL compile failed for " + sourceName, t, Collections.emptySet());
        }
        // All DDL for this file's metrics is now fired. If the opt deferred its schema fence
        // (batched apply via withSchemaChangeDeferredFence), run the single barrier here so the
        // whole file waits ONCE instead of one fence per metric/downsampling. A fence timeout is
        // a non-fatal WARN inside the closure; only a barrier transport error throws, which
        // aborts this apply exactly as an inline per-resource fence would have.
        //
        // EXCEPTION: when fenceRunByCaller is set (the runtime-rule REST apply), the orchestrator
        // runs the fence itself AFTER the durable commit + peer resume, on a background thread, so
        // a long (3-min) cluster-propagation wait neither blocks the apply nor holds peers
        // suspended. We only fire the DDL here and leave the closure for the caller to run.
        if (!storageOpt.isFenceRunByCaller()) {
            try {
                storageOpt.runDeferredFence();
            } catch (final StorageException e) {
                layerRegistry.rollback(appliedClaims);
                throw new ApplyException("schema fence failed for " + sourceName, e, metricNames);
            }
        }
        return new Applied(rule, convert, metricNames, ruleLoader, appliedClaims);
    }

    /** Split {@code "catalog/name"} → catalog half. Falls back to {@code otel-rules} when the
     *  source name is bare (legacy callers, tests). */
    private static String deriveCatalog(final String sourceName) {
        final int idx = sourceName.indexOf('/');
        return idx > 0 ? sourceName.substring(0, idx) : "otel-rules";
    }

    private static String deriveRuleName(final String sourceName) {
        final int idx = sourceName.indexOf('/');
        return idx > 0 ? sourceName.substring(idx + 1) : sourceName;
    }

    /**

View on GitHub (pinned to 102af09b4a)

Solutions

1. Check BanyanDB/storage cluster health and OAP→storage connectivity, then retry the apply; the fence is transport-bound, so transient failures resolve with the connection
2. Increase the fence/propagation timeout if the cluster routinely needs longer than the configured 3-minute window (schema propagation across many data nodes)
3. If using the REST runtime-rule apply path, confirm whether the orchestrator runs the fence (fenceRunByCaller) — in that mode this error moves to the post-commit background fence instead
4. Verify data-node clocks/health: mod_revision is now a client-stamped UnixNano timestamp, so skewed or stalled nodes never reach the watermark

if (!storageClusterHealthy()) deferApply("storage fence target unavailable"); // check schema-role reachability before firing DDL+fence

catch (ApplyException e) when 'schema fence failed': the apply already rolled back layer claims and returned metricNames for rollback — run that rollback, then retry the whole apply once storage connectivity is restored. Retry is safe because the barrier is transport-bound.

When it happens

Trigger: Applying a MAL file through the batched/deferred-fence path (withSchemaChangeDeferredFence) where the storage barrier's transport to the schema/data nodes fails — e.g. BanyanDB schema-server connection dropped mid-fence, gRPC channel reset, or coordinator unavailable. Fires only when fenceRunByCaller is false; the REST orchestrator path runs its own fence later.

Common situations: BanyanDB restart or network partition between OAP and the schema role during apply; gRPC keepalive timeouts on long schema-propagation waits under load; Applying many MAL files concurrently, saturating the barrier stream

Related errors

• Storage-model cascade failed for metric {}; backend drop did
• Only BanyanDB storage support Trace V2 query now.
• local-cache-verify boot: backend resource '{resourceName}' s
• Load meter analyzer configs failed
• Cannot resolve index access on {simpleName} in def variable

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/c73d9492e067bf5f. Report an issue: GitHub.