{"record":{"id":"a7537fb98a1a8a64","repo":"thanos-io/thanos","slug":"open-tsdb","errorCode":null,"errorMessage":"open TSDB","messagePattern":"open TSDB","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/thanos/rule.go","lineNumber":520,"sourceCode":"\t\tagentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"start remote write agent db\")\n\t\t}\n\t\t// We need to call SetWriteNotified() so that agendDB gets notified about every write.\n\t\t// Without it we fallback to polling, which pulls new samples to write every 15s.\n\t\t// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation\n\t\t// and samples being sent over via remote_write.\n\t\tagentDB.SetWriteNotified(remoteStore)\n\t\tfanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)\n\t\tappendable = fanoutStore\n\t\t// Use a separate queryable to restore the ALERTS firing states.\n\t\t// We cannot use remoteStore directly because it uses remote read for\n\t\t// query. However, remote read is not implemented in Thanos Receiver.\n\t\tqueryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)\n\t} else {\n\t\ttsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, \"component\", \"tsdb\")), reg, tsdbOpts, nil)\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"open TSDB\")\n\t\t}\n\n\t\tlevel.Debug(logger).Log(\"msg\", \"removing storage lock file if any\")\n\t\tif err := removeLockfileIfAny(logger, conf.dataDir); err != nil {\n\t\t\treturn errors.Wrap(err, \"remove storage lock files\")\n\t\t}\n\n\t\t{\n\t\t\tdone := make(chan struct{})\n\t\t\tg.Add(func() error {\n\t\t\t\t<-done\n\t\t\t\treturn tsdbDB.Close()\n\t\t\t}, func(error) {\n\t\t\t\tclose(done)\n\t\t\t})\n\t\t}\n\t\tappendable = tsdbDB\n\t\tqueryable = tsdbDB","sourceCodeStart":502,"sourceCodeEnd":538,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/rule.go#L502-L538","documentation":"This error wraps a failure from tsdb.Open when starting the local Prometheus TSDB at conf.dataDir in the non-remote-write path of runRule. It means the TSDB storage engine could not be opened, typically due to data directory issues or invalid TSDB options. Unlike the agent path, this opens a full TSDB for local rule evaluation.","triggerScenarios":"tsdb.Open(conf.dataDir, logger, reg, tsdbOpts, nil) returns an error — directory not writable, corrupt/incompatible persisted blocks or WAL, invalid tsdbOpts (e.g. bad retention or retention size values), or a lock held by another process.","commonSituations":"Disk full during prior run leaving corrupt WAL, upgrading Thanos across incompatible TSDB versions, two rule pods on the same dataDir PVC, retention flags with negative or malformed durations.","solutions":["Read the wrapped error: if it reports a corrupt WAL/checkpoint, back up and remove the wal/ and chunks_head/ dirs under conf.dataDir.","Verify conf.dataDir is writable and exclusively owned by this process (no shared PVC across replicas).","Check tsdb-related flags (--tsdb.retention, retention size, block ranges) are positive and well-formed.","Ensure sufficient disk space and inode availability on the data volume."],"exampleFix":"// before\n--tsdb.retention=-1d  # invalid\n// after\n--tsdb.retention=48h","handlingStrategy":"try-catch","validationCode":"if retention <= 0 {\n    return fmt.Errorf(\"--tsdb.retention must be positive\")\n}\nif err := os.MkdirAll(dataDir, 0o777); err != nil {\n    return fmt.Errorf(\"tsdb dir unusable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"tsdbDB, err = tsdb.Open(dataDir, logger, reg, tsdbOpts, nil)\nif err != nil {\n    if strings.Contains(err.Error(), \"corrupt\") || strings.Contains(err.Error(), \"WAL\") {\n        logger.Error(\"tsdb open failed on storage state; inspect wal/ under data-dir\", \"err\", err)\n    }\n    return errors.Wrap(err, \"open TSDB\")\n}","preventionTips":["Never share a TSDB dataDir between processes or replicas.","Validate retention/range flags before rollout.","Watch disk pressure metrics; corrupt WALs usually follow ENOSPC."],"tags":["thanos","tsdb","storage","startup"],"backgroundTag":"database-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}