{"record":{"id":"95516e76ccc20980","repo":"thanos-io/thanos","slug":"remove-storage-lock-files-95516e","errorCode":null,"errorMessage":"remove storage lock files","messagePattern":"remove storage lock files","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/rule.go","lineNumber":525,"sourceCode":"\t\t// Without it we fallback to polling, which pulls new samples to write every 15s.\n\t\t// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation\n\t\t// and samples being sent over via remote_write.\n\t\tagentDB.SetWriteNotified(remoteStore)\n\t\tfanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)\n\t\tappendable = fanoutStore\n\t\t// Use a separate queryable to restore the ALERTS firing states.\n\t\t// We cannot use remoteStore directly because it uses remote read for\n\t\t// query. However, remote read is not implemented in Thanos Receiver.\n\t\tqueryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)\n\t} else {\n\t\ttsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, \"component\", \"tsdb\")), reg, tsdbOpts, nil)\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"open TSDB\")\n\t\t}\n\n\t\tlevel.Debug(logger).Log(\"msg\", \"removing storage lock file if any\")\n\t\tif err := removeLockfileIfAny(logger, conf.dataDir); err != nil {\n\t\t\treturn errors.Wrap(err, \"remove storage lock files\")\n\t\t}\n\n\t\t{\n\t\t\tdone := make(chan struct{})\n\t\t\tg.Add(func() error {\n\t\t\t\t<-done\n\t\t\t\treturn tsdbDB.Close()\n\t\t\t}, func(error) {\n\t\t\t\tclose(done)\n\t\t\t})\n\t\t}\n\t\tappendable = tsdbDB\n\t\tqueryable = tsdbDB\n\t}\n\n\t// Build the Alertmanager clients.\n\tvar alertingCfg alert.AlertingConfig\n\tif len(conf.alertmgrsConfigYAML) > 0 {","sourceCodeStart":507,"sourceCodeEnd":543,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/rule.go#L507-L543","documentation":"runRule removes any leftover Prometheus TSDB lock file from the rule evaluator's data directory (--data-dir) before opening TSDB, because a previous unclean shutdown can leave the lock behind and block startup. If removeLockfileIfAny fails (e.g. permission or I/O error deleting the file), the error is wrapped as \"remove storage lock files\" and aborts the rule component startup.","triggerScenarios":"Starting `thanos rule` when the dataDir contains a lock file that cannot be removed: filesystem permissions deny unlink, the path is a directory instead of a file, read-only mount, or an I/O error while os.Remove runs.","commonSituations":"Running the container as a non-root user against a persistent volume owned by root; a stale Prometheus lock file left by a crashed previous process on a read-only-mounted PVC; dataDir accidentally pointing at a system directory.","solutions":["Fix ownership/permissions of the --data-dir directory so the thanos process user can delete files in it (chown/chmod).","Check the dataDir path is a writable regular directory, not a file or read-only mount.","Manually remove the stale lock file (e.g. rm <data-dir>/lock) and restart.","If on Kubernetes, ensure the volume's securityContext (fsGroup/runAsUser) matches the container user."],"exampleFix":"// before\nthanos rule --data-dir=/var/thanos/rule   # root-owned dir, container runs as nobody\n// after\nchown -R 65534:65534 /var/thanos/rule && thanos rule --data-dir=/var/thanos/rule","handlingStrategy":"validation","validationCode":"// before starting, as the process user:\nos.Stat(dataDir) and ensure it is a directory with write permission; test -w \"$DATA_DIR\" || { chown -R \"$UID:$GID\" \"$DATA_DIR\"; }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pre-create and chown the dataDir in your entrypoint/init container","Set matching securityContext fsGroup/runAsUser in Kubernetes","Never mount the dataDir volume read-only"],"tags":["filesystem","permissions","startup","tsdb"],"backgroundTag":"file-write-permission-denied","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}