thanos-io/thanos · critical

start remote write agent db

Error message

start remote write agent db

What it means

This error wraps a failure from agent.Open, which opens the embedded Prometheus agent TSDB used as the remote write target store in runRule. It is returned after ApplyConfig succeeds. The agent DB could not be created or opened at conf.dataDir, usually due to filesystem problems or corrupt WAL state.

Solutions

1. Check conf.dataDir exists, is writable by the process user, and is not shared with another replica.
2. Inspect the wrapped error: if the WAL is corrupt, back up and clear the data directory contents.
3. Verify no stale lockfile from a crashed process remains; fix storage permissions after securityContext/fsGroup changes.
4. Confirm disk has free space and the volume is mounted before startup.

Example fix

// before
securityContext:
  runAsUser: 65534  # cannot write to data-dir owned by 0
// after
securityContext:
  runAsUser: 1001
  fsGroup: 1001

if err := os.MkdirAll(dataDir, 0o777); err != nil {
    return fmt.Errorf("data dir unusable: %w", err)
}
f, err := os.CreateTemp(dataDir, "probe")
if err != nil {
    return fmt.Errorf("data dir not writable: %w", err)
}
f.Close(); os.Remove(f.Name())

agentDB, err = agent.Open(slogger, reg, remoteStore, dataDir, agentOpts)
if err != nil {
    if errors.Is(err, os.ErrPermission) || strings.Contains(err.Error(), "lock") {
        logger.Error("agent db open blocked by fs/lock", "dir", dataDir, "err", err)
    }
    return errors.Wrap(err, "start remote write agent db")
}

When it happens

Trigger: agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts) errors — data dir cannot be created/written, contains a corrupt or incompatible WAL, or another instance holds a lock on it.

Common situations: Conf.dataDir volume not writable by the container user (common after securityContext changes), leftover WAL from a crashed pod with a newer/older storage format, or two rule replicas sharing the same persistent volume.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• open TSDB
• get appender
• critical error detected
• failed to parse remote write config
• applying config to remote storage

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/478e97a1813e265c. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:504

		slogger := logutil.GoKitLogToSlog(logger)
		// flushDeadline is set to 1m, but it is for metadata watcher only so not used here.
		// TODO: add type and unit labels support?
		remoteStore := remote.NewStorage(slogger, reg, func() (int64, error) {
			return 0, nil
		}, conf.dataDir, 1*time.Minute, &readyScrapeManager{}, false)
		if err := remoteStore.ApplyConfig(&config.Config{
			GlobalConfig: config.GlobalConfig{
				ExternalLabels: labelsTSDBToProm(conf.lset),
			},
			RemoteWriteConfigs: rwCfg.RemoteWriteConfigs,
		}); err != nil {
			return errors.Wrap(err, "applying config to remote storage")
		}

		agentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)
		if err != nil {
			return errors.Wrap(err, "start remote write agent db")
		}
		// We need to call SetWriteNotified() so that agendDB gets notified about every write.
		// Without it we fallback to polling, which pulls new samples to write every 15s.
		// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation
		// and samples being sent over via remote_write.
		agentDB.SetWriteNotified(remoteStore)
		fanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)
		appendable = fanoutStore
		// Use a separate queryable to restore the ALERTS firing states.
		// We cannot use remoteStore directly because it uses remote read for
		// query. However, remote read is not implemented in Thanos Receiver.
		queryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)
	} else {
		tsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, "component", "tsdb")), reg, tsdbOpts, nil)
		if err != nil {
			return errors.Wrap(err, "open TSDB")
		}

View on GitHub (pinned to 35b8b99117)