apache/skywalking · error · UnexpectedException

Load rule file {} failed

Error message

Load rule file {} failed

What it means

UnexpectedException ('Load rule file X failed') thrown while the Rules loader walks the MAL rules directory and Files.readAllBytes raises an IOException for one rule file. It means the file was visible to the directory walker but could not be read — permissions changed, the file was deleted mid-walk, or the filesystem raised an I/O error.

Source

Thrown at oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java:125

                }
                return formedEnabledRules.keySet().stream().anyMatch(rule -> {
                    boolean matches = FileSystems.getDefault().getPathMatcher("glob:" + rule)
                        .matches(root.relativize(p));
                    if (matches) {
                        formedEnabledRules.put(rule, true);
                    }
                    return matches;
                });
            }).forEach(p -> {
                final String rel = root.relativize(p).toString();
                final String ruleName = rel.substring(0, rel.lastIndexOf('.'));
                // Keep the actual extension: both .yaml and .yml load, and synthesising .yaml for
                // a .yml rule points the provenance at a file that does not exist.
                diskPaths.put(ruleName, rel);
                try {
                    diskBytes.put(ruleName, Files.readAllBytes(p));
                } catch (IOException e) {
                    throw new UnexpectedException("Load rule file " + p.getFileName() + " failed", e);
                }
            });
        }

        if (formedEnabledRules.containsValue(false)) {
            List<String> rulesNotFound = formedEnabledRules.keySet().stream()
                    .filter(rule -> !formedEnabledRules.get(rule))
                    .collect(Collectors.toList());
            throw new UnexpectedException("Some configuration files of enabled rules are not found, enabled rules: " + rulesNotFound);
        }

        // Merge with classpath-discovered resolvers (runtime-rule DB, plus any future
        // priority-ranked source). ACTIVE substitutes existing disk entries; INACTIVE
        // drops them. Resolver-only rules (no disk twin) are NOT merged here — they're
        // applied by RuleSync.runOnce post-seal via the dynamic layer channel.
        final Map<String, byte[]> merged = useInstalledManager
            ? RuleSetMerger.merge(path, diskBytes)
            : RuleSetMerger.merge(path, diskBytes, manager);

View on GitHub (pinned to 102af09b4a)

Solutions

1. Check file permissions: the OAP process user needs read access to every .yaml/.yml under the rules directory
2. Verify filesystem health and that no process is mutating rule files during OAP startup (stagger config-sync sidecars vs OAP boot)
3. If the file was intentionally deleted, remove any stale reference to it from the enabled-rules list and restart
4. Retry OAP startup after fixing the mount/permission; the exception aborts rule loading so a restart is required

// Before OAP start: verify readability of every rule file
try (Stream<Path> s = Files.walk(rulesPath)) {
    for (Path p : (Iterable<Path>) s.filter(f -> f.toString().endsWith(".yaml") || f.toString().endsWith(".yml"))::iterator) {
        if (!Files.isReadable(p)) throw new IllegalStateException("Not readable: " + p);
    }
}

catch (UnexpectedException e) { /* check e.getCause() instanceof IOException; fix mount/permissions and restart OAP */ }

When it happens

Trigger: An IOException during Files.readAllBytes(p) for a rule file under the meter-analyzer rules path (rulesPath config). Typical on containers where the rules directory is bind-mounted read-only with odd permissions, or when a file is removed/truncated between directory listing and read.

Common situations: Kubernetes volume mounts with restrictive fileMode; sidecars (config rewriters, secret injectors) atomically replacing rule files while OAP boots; NFS/FUSE-backed mounts returning transient I/O errors; disk-full conditions.

Related errors

• Some configuration files of enabled rules are not found, ena
• Load rule {} failed
• MAL rule {} layerDefinitions entry rejected: {}
• Failed to load MAL rules
• decorate() should be invoked after service()

AI-assisted analysis of apache/skywalking@102af09b4a (2026-08-14). Data as JSON: /api/errors/136d820b0a284188. Report an issue: GitHub.