thanos-io/thanos · error

applying config to remote storage

Error message

applying config to remote storage

What it means

This error wraps a failure from remoteStore.ApplyConfig when pushing the parsed remote write configs plus external labels into the remote storage WAL/queue manager. runRule calls it right after parsing the YAML. ApplyConfig rejects configs that Prometheus's remote write implementation considers invalid.

Solutions

1. Ensure each remote_write entry has a unique `name` if more than one is configured.
2. Validate the remote_write url (scheme + host) is well-formed and reachable.
3. Compare against a working prometheus.yml remote_write section using promtool.
4. Read the wrapped error for the specific config index/field ApplyConfig rejected.

Example fix

// before
remote_write:
  - url: http://r1/api/v1/receive
    name: rw
  - url: http://r2/api/v1/receive
    name: rw   # duplicate
// after
remote_write:
  - url: http://r1/api/v1/receive
    name: rw-1
  - url: http://r2/api/v1/receive
    name: rw-2

names := map[string]bool{}
for _, c := range rwCfg.RemoteWriteConfigs {
    if c.Name != "" {
        if names[c.Name] {
            return fmt.Errorf("duplicate remote_write name %q", c.Name)
        }
        names[c.Name] = true
    }
    if _, err := url.Parse(c.URLConfig.URL.String()); err != nil {
        return fmt.Errorf("bad remote_write url: %w", err)
    }
}

if err := remoteStore.ApplyConfig(cfg); err != nil {
    logger.Error("remote write ApplyConfig rejected config", "err", err)
    return errors.Wrap(err, "applying config to remote storage")
}

When it happens

Trigger: remoteStore.ApplyConfig(&config.Config{...}) returns an error — e.g. duplicate remote_write queue names, invalid url in a remote_write config, or zero/misconfigured configs at the storage layer.

Common situations: Two remote_write entries with the same name, an invalid or unreachable url scheme, or external labels conflicting with what ApplyConfig validates.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• unable to load config file
• unable to create config reloader
• unable to load config initially
• tracing failed
• query configuration

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/77defef3677b7faf. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:499

			RemoteWriteConfigs []*config.RemoteWriteConfig `yaml:"remote_write,omitempty"`
		}
		if err := yaml.Unmarshal(rwCfgYAML, &rwCfg); err != nil {
			return errors.Wrapf(err, "failed to parse remote write config %v", string(rwCfgYAML))
		}

		slogger := logutil.GoKitLogToSlog(logger)
		// flushDeadline is set to 1m, but it is for metadata watcher only so not used here.
		// TODO: add type and unit labels support?
		remoteStore := remote.NewStorage(slogger, reg, func() (int64, error) {
			return 0, nil
		}, conf.dataDir, 1*time.Minute, &readyScrapeManager{}, false)
		if err := remoteStore.ApplyConfig(&config.Config{
			GlobalConfig: config.GlobalConfig{
				ExternalLabels: labelsTSDBToProm(conf.lset),
			},
			RemoteWriteConfigs: rwCfg.RemoteWriteConfigs,
		}); err != nil {
			return errors.Wrap(err, "applying config to remote storage")
		}

		agentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)
		if err != nil {
			return errors.Wrap(err, "start remote write agent db")
		}
		// We need to call SetWriteNotified() so that agendDB gets notified about every write.
		// Without it we fallback to polling, which pulls new samples to write every 15s.
		// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation
		// and samples being sent over via remote_write.
		agentDB.SetWriteNotified(remoteStore)
		fanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)
		appendable = fanoutStore
		// Use a separate queryable to restore the ALERTS firing states.
		// We cannot use remoteStore directly because it uses remote read for
		// query. However, remote read is not implemented in Thanos Receiver.
		queryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)
	} else {

View on GitHub (pinned to 35b8b99117)