thanos-io/thanos · critical

open TSDB

Error message

open TSDB

What it means

This error wraps a failure from tsdb.Open when starting the local Prometheus TSDB at conf.dataDir in the non-remote-write path of runRule. It means the TSDB storage engine could not be opened, typically due to data directory issues or invalid TSDB options. Unlike the agent path, this opens a full TSDB for local rule evaluation.

Solutions

1. Read the wrapped error: if it reports a corrupt WAL/checkpoint, back up and remove the wal/ and chunks_head/ dirs under conf.dataDir.
2. Verify conf.dataDir is writable and exclusively owned by this process (no shared PVC across replicas).
3. Check tsdb-related flags (--tsdb.retention, retention size, block ranges) are positive and well-formed.
4. Ensure sufficient disk space and inode availability on the data volume.

Example fix

// before
--tsdb.retention=-1d  # invalid
// after
--tsdb.retention=48h

if retention <= 0 {
    return fmt.Errorf("--tsdb.retention must be positive")
}
if err := os.MkdirAll(dataDir, 0o777); err != nil {
    return fmt.Errorf("tsdb dir unusable: %w", err)
}

tsdbDB, err = tsdb.Open(dataDir, logger, reg, tsdbOpts, nil)
if err != nil {
    if strings.Contains(err.Error(), "corrupt") || strings.Contains(err.Error(), "WAL") {
        logger.Error("tsdb open failed on storage state; inspect wal/ under data-dir", "err", err)
    }
    return errors.Wrap(err, "open TSDB")
}

When it happens

Trigger: tsdb.Open(conf.dataDir, logger, reg, tsdbOpts, nil) returns an error — directory not writable, corrupt/incompatible persisted blocks or WAL, invalid tsdbOpts (e.g. bad retention or retention size values), or a lock held by another process.

Common situations: Disk full during prior run leaving corrupt WAL, upgrading Thanos across incompatible TSDB versions, two rule pods on the same dataDir PVC, retention flags with negative or malformed durations.

Related errors

• start remote write agent db
• get appender
• create compactor
• create bucket compactor
• critical error detected

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/a7537fb98a1a8a64. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:520

		agentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)
		if err != nil {
			return errors.Wrap(err, "start remote write agent db")
		}
		// We need to call SetWriteNotified() so that agendDB gets notified about every write.
		// Without it we fallback to polling, which pulls new samples to write every 15s.
		// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation
		// and samples being sent over via remote_write.
		agentDB.SetWriteNotified(remoteStore)
		fanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)
		appendable = fanoutStore
		// Use a separate queryable to restore the ALERTS firing states.
		// We cannot use remoteStore directly because it uses remote read for
		// query. However, remote read is not implemented in Thanos Receiver.
		queryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)
	} else {
		tsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, "component", "tsdb")), reg, tsdbOpts, nil)
		if err != nil {
			return errors.Wrap(err, "open TSDB")
		}

		level.Debug(logger).Log("msg", "removing storage lock file if any")
		if err := removeLockfileIfAny(logger, conf.dataDir); err != nil {
			return errors.Wrap(err, "remove storage lock files")
		}

		{
			done := make(chan struct{})
			g.Add(func() error {
				<-done
				return tsdbDB.Close()
			}, func(error) {
				close(done)
			})
		}
		appendable = tsdbDB
		queryable = tsdbDB

View on GitHub (pinned to 35b8b99117)