thanos-io/thanos · error

failed to parse remote write config

Error message

failed to parse remote write config %v

What it means

This error wraps a yaml.Unmarshal failure when parsing the remote write configuration YAML (conf.remoteWrite.config) into the {remote_write: [...]} struct in runRule. errors.Wrapf includes the raw YAML to help spot the problem. It means the provided remote-write YAML does not match Prometheus's remote_write schema.

Solutions

1. Validate the YAML parses standalone (yamlted or `promtool check config` on the equivalent Prometheus config).
2. Ensure the YAML has a top-level `remote_write:` key with a list of valid Prometheus remote_write configs — not a full prometheus.yml.
3. Remove fields unsupported by the vendored Prometheus config version in this Thanos build.
4. Check env substitution didn't inject empty or malformed values into the YAML.

Example fix

// before
# passed file contains full prometheus.yml
scrape_configs: ...
// after
remote_write:
  - url: http://receive:19291/api/v1/receive
    queue_config:
      max_samples_per_send: 2000

var probe struct {
    RemoteWriteConfigs []*config.RemoteWriteConfig `yaml:"remote_write"`
}
if err := yaml.UnmarshalStrict(rwCfgYAML, &probe); err != nil {
    return fmt.Errorf("remote_write yaml invalid: %w", err)
}

When it happens

Trigger: len(rwCfgYAML) > 0 and yaml.Unmarshal(rwCfgYAML, &rwCfg) fails — invalid YAML syntax or fields not present in config.RemoteWriteConfig.

Common situations: Using Thanos-incompatible remote_write fields, YAML indentation errors, passing a full Prometheus config file instead of just the remote_write section, quoting issues from env-var substitution in manifests.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• unable to unmarshal config content
• query configuration
• applying config to remote storage
• parsing http config YAML
• unable to parse objstore config

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/552c8cac646d8eec. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:484

	var (
		appendable storage.Appendable
		queryable  storage.Queryable
		tsdbDB     *tsdb.DB
		agentDB    *agent.DB
	)

	rwCfgYAML, err := conf.rwConfig.Content()
	if err != nil {
		return err
	}

	if len(rwCfgYAML) > 0 {
		var rwCfg struct {
			RemoteWriteConfigs []*config.RemoteWriteConfig `yaml:"remote_write,omitempty"`
		}
		if err := yaml.Unmarshal(rwCfgYAML, &rwCfg); err != nil {
			return errors.Wrapf(err, "failed to parse remote write config %v", string(rwCfgYAML))
		}

		slogger := logutil.GoKitLogToSlog(logger)
		// flushDeadline is set to 1m, but it is for metadata watcher only so not used here.
		// TODO: add type and unit labels support?
		remoteStore := remote.NewStorage(slogger, reg, func() (int64, error) {
			return 0, nil
		}, conf.dataDir, 1*time.Minute, &readyScrapeManager{}, false)
		if err := remoteStore.ApplyConfig(&config.Config{
			GlobalConfig: config.GlobalConfig{
				ExternalLabels: labelsTSDBToProm(conf.lset),
			},
			RemoteWriteConfigs: rwCfg.RemoteWriteConfigs,
		}); err != nil {
			return errors.Wrap(err, "applying config to remote storage")
		}

		agentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)

View on GitHub (pinned to 35b8b99117)