{"record":{"id":"c73d9492e067bf5f","repo":"apache/skywalking","slug":"schema-fence-failed-for-sourcename","errorCode":null,"errorMessage":"schema fence failed for {sourceName}","messagePattern":"schema fence failed for (.+?)","errorType":"http","errorClass":"ApplyException","httpStatus":500,"severity":"error","filePath":"oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java","lineNumber":189,"sourceCode":"            layerRegistry.rollback(appliedClaims);\n            throw new ApplyException(\"MAL compile failed for \" + sourceName, t, Collections.emptySet());\n        }\n        // All DDL for this file's metrics is now fired. If the opt deferred its schema fence\n        // (batched apply via withSchemaChangeDeferredFence), run the single barrier here so the\n        // whole file waits ONCE instead of one fence per metric/downsampling. A fence timeout is\n        // a non-fatal WARN inside the closure; only a barrier transport error throws, which\n        // aborts this apply exactly as an inline per-resource fence would have.\n        //\n        // EXCEPTION: when fenceRunByCaller is set (the runtime-rule REST apply), the orchestrator\n        // runs the fence itself AFTER the durable commit + peer resume, on a background thread, so\n        // a long (3-min) cluster-propagation wait neither blocks the apply nor holds peers\n        // suspended. We only fire the DDL here and leave the closure for the caller to run.\n        if (!storageOpt.isFenceRunByCaller()) {\n            try {\n                storageOpt.runDeferredFence();\n            } catch (final StorageException e) {\n                layerRegistry.rollback(appliedClaims);\n                throw new ApplyException(\"schema fence failed for \" + sourceName, e, metricNames);\n            }\n        }\n        return new Applied(rule, convert, metricNames, ruleLoader, appliedClaims);\n    }\n\n    /** Split {@code \"catalog/name\"} → catalog half. Falls back to {@code otel-rules} when the\n     *  source name is bare (legacy callers, tests). */\n    private static String deriveCatalog(final String sourceName) {\n        final int idx = sourceName.indexOf('/');\n        return idx > 0 ? sourceName.substring(0, idx) : \"otel-rules\";\n    }\n\n    private static String deriveRuleName(final String sourceName) {\n        final int idx = sourceName.indexOf('/');\n        return idx > 0 ? sourceName.substring(idx + 1) : sourceName;\n    }\n\n    /**","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/apache/skywalking/blob/102af09b4a56064e22050dded10e2c52e490d040/oap-server/server-admin/runtime-rule/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/apply/MalFileApplier.java#L171-L207","documentation":"ApplyException thrown when storageOpt.runDeferredFence() raises StorageException after all metric DDL for the file was fired. The deferred schema fence is the single barrier that waits for schema propagation (e.g. BanyanDB mod_revision watermark across data nodes); only a barrier transport error throws — a mere timeout is a non-fatal WARN inside the closure. On throw, layer claims are rolled back and metricNames are passed as the rollback set.","triggerScenarios":"Applying a MAL file through the batched/deferred-fence path (withSchemaChangeDeferredFence) where the storage barrier's transport to the schema/data nodes fails — e.g. BanyanDB schema-server connection dropped mid-fence, gRPC channel reset, or coordinator unavailable. Fires only when fenceRunByCaller is false; the REST orchestrator path runs its own fence later.","commonSituations":"BanyanDB restart or network partition between OAP and the schema role during apply; gRPC keepalive timeouts on long schema-propagation waits under load; Applying many MAL files concurrently, saturating the barrier stream","solutions":["Check BanyanDB/storage cluster health and OAP→storage connectivity, then retry the apply; the fence is transport-bound, so transient failures resolve with the connection","Increase the fence/propagation timeout if the cluster routinely needs longer than the configured 3-minute window (schema propagation across many data nodes)","If using the REST runtime-rule apply path, confirm whether the orchestrator runs the fence (fenceRunByCaller) — in that mode this error moves to the post-commit background fence instead","Verify data-node clocks/health: mod_revision is now a client-stamped UnixNano timestamp, so skewed or stalled nodes never reach the watermark"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if (!storageClusterHealthy()) deferApply(\"storage fence target unavailable\"); // check schema-role reachability before firing DDL+fence","typeGuard":null,"tryCatchPattern":"catch (ApplyException e) when 'schema fence failed': the apply already rolled back layer claims and returned metricNames for rollback — run that rollback, then retry the whole apply once storage connectivity is restored. Retry is safe because the barrier is transport-bound.","preventionTips":["Monitor OAP→BanyanDB schema-role channel health before scheduling rule pushes","Size the fence timeout to your largest expected propagation window (many data nodes → longer)","Prefer the REST orchestrator path (fenceRunByCaller) for batch pushes so a long fence doesn't hold peers suspended"],"tags":["storage","banyandb","schema-fence","mal"],"backgroundTag":null,"analyzedSha":"102af09b4a56064e22050dded10e2c52e490d040","analyzedAt":"2026-08-14T10:47:52.647Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}