thanos-io/thanos · error

remove storage lock files

Error message

remove storage lock files

What it means

runRule removes any leftover Prometheus TSDB lock file from the rule evaluator's data directory (--data-dir) before opening TSDB, because a previous unclean shutdown can leave the lock behind and block startup. If removeLockfileIfAny fails (e.g. permission or I/O error deleting the file), the error is wrapped as "remove storage lock files" and aborts the rule component startup.

Solutions

1. Fix ownership/permissions of the --data-dir directory so the thanos process user can delete files in it (chown/chmod).
2. Check the dataDir path is a writable regular directory, not a file or read-only mount.
3. Manually remove the stale lock file (e.g. rm <data-dir>/lock) and restart.
4. If on Kubernetes, ensure the volume's securityContext (fsGroup/runAsUser) matches the container user.

Example fix

// before
thanos rule --data-dir=/var/thanos/rule   # root-owned dir, container runs as nobody
// after
chown -R 65534:65534 /var/thanos/rule && thanos rule --data-dir=/var/thanos/rule

// before starting, as the process user:
os.Stat(dataDir) and ensure it is a directory with write permission; test -w "$DATA_DIR" || { chown -R "$UID:$GID" "$DATA_DIR"; }

When it happens

Trigger: Starting `thanos rule` when the dataDir contains a lock file that cannot be removed: filesystem permissions deny unlink, the path is a directory instead of a file, read-only mount, or an I/O error while os.Remove runs.

Common situations: Running the container as a non-root user against a persistent volume owned by root; a stale Prometheus lock file left by a crashed previous process on a read-only-mounted PVC; dataDir accidentally pointing at a system directory.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• create working compact directory
• create working downsample directory
• create default tenant data dir
• create meta fetcher
• create compactor

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/95516e76ccc20980. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/rule.go:525

		// Without it we fallback to polling, which pulls new samples to write every 15s.
		// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation
		// and samples being sent over via remote_write.
		agentDB.SetWriteNotified(remoteStore)
		fanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)
		appendable = fanoutStore
		// Use a separate queryable to restore the ALERTS firing states.
		// We cannot use remoteStore directly because it uses remote read for
		// query. However, remote read is not implemented in Thanos Receiver.
		queryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)
	} else {
		tsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, "component", "tsdb")), reg, tsdbOpts, nil)
		if err != nil {
			return errors.Wrap(err, "open TSDB")
		}

		level.Debug(logger).Log("msg", "removing storage lock file if any")
		if err := removeLockfileIfAny(logger, conf.dataDir); err != nil {
			return errors.Wrap(err, "remove storage lock files")
		}

		{
			done := make(chan struct{})
			g.Add(func() error {
				<-done
				return tsdbDB.Close()
			}, func(error) {
				close(done)
			})
		}
		appendable = tsdbDB
		queryable = tsdbDB
	}

	// Build the Alertmanager clients.
	var alertingCfg alert.AlertingConfig
	if len(conf.alertmgrsConfigYAML) > 0 {

View on GitHub (pinned to 35b8b99117)