error unmarshaling query rules: %v, original data '%s' versi

Error message

error unmarshaling query rules: %v, original data '%s' version %v

What it means

topocustomrule watches a topo path holding QueryRules JSON for the tablet's query service. When a watch event fires, apply() unmarshals the watched contents into rules.Rules; if the stored JSON is not valid query-rules JSON, the watch update is rejected with this error carrying the unmarshal error, the raw data, and the topo version — leaving the previous rules in effect.

Source

Thrown at go/vt/vttablet/customrule/topocustomrule/topocustomrule.go:134

			log.Warn(fmt.Sprintf("Sleeping for %v before trying again", sleepDuringTopoFailure))
			time.Sleep(sleepDuringTopoFailure)
		}
	}()
}

func (cr *topoCustomRule) stop() {
	cr.mu.Lock()
	if cr.cancel != nil {
		cr.cancel()
	}
	cr.stopped = true
	cr.mu.Unlock()
}

func (cr *topoCustomRule) apply(wd *topo.WatchData) error {
	qrs := rules.New()
	if err := qrs.UnmarshalJSON(wd.Contents); err != nil {
		return fmt.Errorf("error unmarshaling query rules: %v, original data '%s' version %v", err, wd.Contents, wd.Version)
	}

	if !reflect.DeepEqual(cr.qrs, qrs) {
		cr.qrs = qrs.Copy()
		cr.qsc.SetQueryRules(topoCustomRuleSource, qrs)
		log.Info(fmt.Sprintf("Custom rule version %v fetched from topo and applied to vttablet", wd.Version))
	}

	return nil
}

func (cr *topoCustomRule) oneWatch() error {
	defer func() {
		// Whatever happens, cancel() won't be valid after this function exits.
		cr.mu.Lock()
		cr.cancel = nil
		cr.mu.Unlock()
	}()

View on GitHub (pinned to 01a25a7d17)

Solutions

Fix the JSON stored at the topo custom rules path to be a valid QueryRules document (validate with jq or rules.New().UnmarshalJSON in a test)
Re-write the file via vtctldclient so a fresh version triggers a new watch event and the rules get applied
Check the 'original data' and 'version' embedded in the error to find the exact topo entry to correct
Confirm you are editing the path this vttablet watches (topoCustomRuleSource) and not another cell's copy

Example fix

// topo file before (invalid)
{"rules": [{"match": "SELECT *"}]}
// topo file after (valid QueryRules)
{"rules": [{"description": "block", "match": "SELECT .* FROM t", "action": "FAIL"}]}

Defensive patterns

Strategy: validation

Validate before calling

var probe rules.Rules
if err := json.Unmarshal(contents, &probe); err != nil {
    return fmt.Errorf("custom rules JSON invalid: %v", err)
}

Try / catch

if err := applyRules(wd); err != nil && strings.Contains(err.Error(), "error unmarshaling query rules") {
    log.Error("bad topo custom rules; keeping previous rules", slog.Any("error", err))
}

Prevention

Validate JSON with jq before writing to the topo path
Use vtctldclient to write custom rules instead of hand-editing topo
Confirm version parity: apply once with UnmarshalJSON in CI before rollout

When it happens

Trigger: oneWatch receives a WatchData whose Contents fail rules.Rules.UnmarshalJSON — i.e. the value written to the topo custom-rule path is malformed JSON, a wrong schema, or a truncated write.

Common situations: Operator wrote hand-edited JSON to the custom rules topo file with a syntax error; wrote the right JSON under the wrong path so the schema doesn't match; a partial topo write during a network blip.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/6a9ca2f3944273fa. Report an issue: GitHub.