thanos-io/thanos · error

reloading rules failed

Error message

reloading rules failed

What it means

After collecting rule files, reloadRules calls ruleMgr.Update(evalInterval, files) to (re)load them into the rule manager. Any parse/validation error inside the rule files (bad YAML, invalid expressions, duplicate rule names) is wrapped as "reloading rules failed", configSuccess is set to 0, and the reload aborts; previously loaded rules keep running.

Solutions

1. Run promtool check rules on all matched files to find the exact parse/validation error.
2. Read the wrapped error in logs — it names the offending file and line.
3. Fix or revert the bad rule file, then trigger the reload again.
4. Add rule-file validation to CI (promtool) so invalid files never reach the reload endpoint.

Example fix

// before
groups:
  - name: example
    rules:
      - record: job:up:sum
        expr: sum(up) BY (job   # unbalanced parens
// after
groups:
  - name: example
    rules:
      - record: job:up:sum
        expr: sum(up) by (job)

promtool check rules /etc/thanos/rules/*.yaml && curl -XPOST http://rule:10902/-/reload

resp, err := http.Post(reloadURL, "", nil)
if err != nil || resp.StatusCode != 200 {
  body, _ := io.ReadAll(resp.Body)
  log.Errorf("reload failed: %s", body) // wrapped 'reloading rules failed' chain
}

When it happens

Trigger: POST /-/reload or SIGHUP (or periodic reload) when one of the matched rule files contains invalid YAML, a syntactically invalid PromQL expression, or fails Prometheus rule validation in rules.ParseGroup/Load.

Common situations: CI deploying a hand-edited rules YAML with indentation errors; a rule referencing a nonexistent template; duplicate group names after merging files; migration between Prometheus versions with changed validation.

Related errors

• unable to unmarshal config content
• retrieving rule files failed. Ignoring file. pattern
• unknown integration name
• parse limit configuration
• Groupname should not be empty

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fe36a23d727ddd79. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:1109

			// The only error can be a bad pattern.
			errs.Add(errors.Wrapf(err, "retrieving rule files failed. Ignoring file. pattern %s", pat))
			continue
		}

		for _, fp := range fs {
			if _, ok := seenFiles[fp]; ok {
				continue
			}
			files = append(files, fp)
			seenFiles[fp] = struct{}{}
		}
	}

	level.Info(logger).Log("msg", "reload rule files", "numFiles", len(files))

	if err := ruleMgr.Update(evalInterval, files); err != nil {
		metrics.configSuccess.Set(0)
		errs.Add(errors.Wrap(err, "reloading rules failed"))
		return errs.Err()
	}

	metrics.configSuccess.Set(1)
	metrics.configSuccessTime.Set(float64(time.Now().UnixNano()) / 1e9)

	metrics.rulesLoaded.Reset()
	for _, group := range ruleMgr.RuleGroups() {
		metrics.rulesLoaded.WithLabelValues(group.PartialResponseStrategy.String(), group.OriginalFile, group.Name()).Set(float64(len(group.Rules())))
	}
	return errs.Err()
}

func tableLinkForExpression(tmpl string, expr string) (string, error) {
	// template example: "/graph?g0.expr={{.Expr}}&g0.tab=1"
	escapedExpression := url.QueryEscape(expr)

	escapedExpr := Expression{Expr: escapedExpression}

View on GitHub (pinned to 35b8b99117)