{"record":{"id":"fe36a23d727ddd79","repo":"thanos-io/thanos","slug":"reloading-rules-failed","errorCode":null,"errorMessage":"reloading rules failed","messagePattern":"reloading rules failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/rule.go","lineNumber":1109,"sourceCode":"\t\t\t// The only error can be a bad pattern.\n\t\t\terrs.Add(errors.Wrapf(err, \"retrieving rule files failed. Ignoring file. pattern %s\", pat))\n\t\t\tcontinue\n\t\t}\n\n\t\tfor _, fp := range fs {\n\t\t\tif _, ok := seenFiles[fp]; ok {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tfiles = append(files, fp)\n\t\t\tseenFiles[fp] = struct{}{}\n\t\t}\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"reload rule files\", \"numFiles\", len(files))\n\n\tif err := ruleMgr.Update(evalInterval, files); err != nil {\n\t\tmetrics.configSuccess.Set(0)\n\t\terrs.Add(errors.Wrap(err, \"reloading rules failed\"))\n\t\treturn errs.Err()\n\t}\n\n\tmetrics.configSuccess.Set(1)\n\tmetrics.configSuccessTime.Set(float64(time.Now().UnixNano()) / 1e9)\n\n\tmetrics.rulesLoaded.Reset()\n\tfor _, group := range ruleMgr.RuleGroups() {\n\t\tmetrics.rulesLoaded.WithLabelValues(group.PartialResponseStrategy.String(), group.OriginalFile, group.Name()).Set(float64(len(group.Rules())))\n\t}\n\treturn errs.Err()\n}\n\nfunc tableLinkForExpression(tmpl string, expr string) (string, error) {\n\t// template example: \"/graph?g0.expr={{.Expr}}&g0.tab=1\"\n\tescapedExpression := url.QueryEscape(expr)\n\n\tescapedExpr := Expression{Expr: escapedExpression}","sourceCodeStart":1091,"sourceCodeEnd":1127,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/rule.go#L1091-L1127","documentation":"After collecting rule files, reloadRules calls ruleMgr.Update(evalInterval, files) to (re)load them into the rule manager. Any parse/validation error inside the rule files (bad YAML, invalid expressions, duplicate rule names) is wrapped as \"reloading rules failed\", configSuccess is set to 0, and the reload aborts; previously loaded rules keep running.","triggerScenarios":"POST /-/reload or SIGHUP (or periodic reload) when one of the matched rule files contains invalid YAML, a syntactically invalid PromQL expression, or fails Prometheus rule validation in rules.ParseGroup/Load.","commonSituations":"CI deploying a hand-edited rules YAML with indentation errors; a rule referencing a nonexistent template; duplicate group names after merging files; migration between Prometheus versions with changed validation.","solutions":["Run promtool check rules on all matched files to find the exact parse/validation error.","Read the wrapped error in logs — it names the offending file and line.","Fix or revert the bad rule file, then trigger the reload again.","Add rule-file validation to CI (promtool) so invalid files never reach the reload endpoint."],"exampleFix":"// before\ngroups:\n  - name: example\n    rules:\n      - record: job:up:sum\n        expr: sum(up) BY (job   # unbalanced parens\n// after\ngroups:\n  - name: example\n    rules:\n      - record: job:up:sum\n        expr: sum(up) by (job)","handlingStrategy":"validation","validationCode":"promtool check rules /etc/thanos/rules/*.yaml && curl -XPOST http://rule:10902/-/reload","typeGuard":null,"tryCatchPattern":"resp, err := http.Post(reloadURL, \"\", nil)\nif err != nil || resp.StatusCode != 200 {\n  body, _ := io.ReadAll(resp.Body)\n  log.Errorf(\"reload failed: %s\", body) // wrapped 'reloading rules failed' chain\n}","preventionTips":["Run promtool check rules in CI on every rules change","Keep group names unique across files","Read the wrapped error in logs for file/line details","Canary reloads against a staging rule component first"],"tags":["rules","yaml","config","validation"],"backgroundTag":"invalid-config-file","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}