{"record":{"id":"478e97a1813e265c","repo":"thanos-io/thanos","slug":"start-remote-write-agent-db","errorCode":null,"errorMessage":"start remote write agent db","messagePattern":"start remote write agent db","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/thanos/rule.go","lineNumber":504,"sourceCode":"\n\t\tslogger := logutil.GoKitLogToSlog(logger)\n\t\t// flushDeadline is set to 1m, but it is for metadata watcher only so not used here.\n\t\t// TODO: add type and unit labels support?\n\t\tremoteStore := remote.NewStorage(slogger, reg, func() (int64, error) {\n\t\t\treturn 0, nil\n\t\t}, conf.dataDir, 1*time.Minute, &readyScrapeManager{}, false)\n\t\tif err := remoteStore.ApplyConfig(&config.Config{\n\t\t\tGlobalConfig: config.GlobalConfig{\n\t\t\t\tExternalLabels: labelsTSDBToProm(conf.lset),\n\t\t\t},\n\t\t\tRemoteWriteConfigs: rwCfg.RemoteWriteConfigs,\n\t\t}); err != nil {\n\t\t\treturn errors.Wrap(err, \"applying config to remote storage\")\n\t\t}\n\n\t\tagentDB, err = agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts)\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"start remote write agent db\")\n\t\t}\n\t\t// We need to call SetWriteNotified() so that agendDB gets notified about every write.\n\t\t// Without it we fallback to polling, which pulls new samples to write every 15s.\n\t\t// If we don't call SetWriteNotified() we'll have up to 15s lag between rule evaluation\n\t\t// and samples being sent over via remote_write.\n\t\tagentDB.SetWriteNotified(remoteStore)\n\t\tfanoutStore := storage.NewFanout(slogger, agentDB, remoteStore)\n\t\tappendable = fanoutStore\n\t\t// Use a separate queryable to restore the ALERTS firing states.\n\t\t// We cannot use remoteStore directly because it uses remote read for\n\t\t// query. However, remote read is not implemented in Thanos Receiver.\n\t\tqueryable = thanosrules.NewPromClientsQueryable(logger, queryClients, promClients, conf.query.httpMethod, conf.query.step, conf.ignoredLabelNames)\n\t} else {\n\t\ttsdbDB, err = tsdb.Open(conf.dataDir, logutil.GoKitLogToSlog(log.With(logger, \"component\", \"tsdb\")), reg, tsdbOpts, nil)\n\t\tif err != nil {\n\t\t\treturn errors.Wrap(err, \"open TSDB\")\n\t\t}\n","sourceCodeStart":486,"sourceCodeEnd":522,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/rule.go#L486-L522","documentation":"This error wraps a failure from agent.Open, which opens the embedded Prometheus agent TSDB used as the remote write target store in runRule. It is returned after ApplyConfig succeeds. The agent DB could not be created or opened at conf.dataDir, usually due to filesystem problems or corrupt WAL state.","triggerScenarios":"agent.Open(slogger, reg, remoteStore, conf.dataDir, agentOpts) errors — data dir cannot be created/written, contains a corrupt or incompatible WAL, or another instance holds a lock on it.","commonSituations":"Conf.dataDir volume not writable by the container user (common after securityContext changes), leftover WAL from a crashed pod with a newer/older storage format, or two rule replicas sharing the same persistent volume.","solutions":["Check conf.dataDir exists, is writable by the process user, and is not shared with another replica.","Inspect the wrapped error: if the WAL is corrupt, back up and clear the data directory contents.","Verify no stale lockfile from a crashed process remains; fix storage permissions after securityContext/fsGroup changes.","Confirm disk has free space and the volume is mounted before startup."],"exampleFix":"// before\nsecurityContext:\n  runAsUser: 65534  # cannot write to data-dir owned by 0\n// after\nsecurityContext:\n  runAsUser: 1001\n  fsGroup: 1001","handlingStrategy":"try-catch","validationCode":"if err := os.MkdirAll(dataDir, 0o777); err != nil {\n    return fmt.Errorf(\"data dir unusable: %w\", err)\n}\nf, err := os.CreateTemp(dataDir, \"probe\")\nif err != nil {\n    return fmt.Errorf(\"data dir not writable: %w\", err)\n}\nf.Close(); os.Remove(f.Name())","typeGuard":null,"tryCatchPattern":"agentDB, err = agent.Open(slogger, reg, remoteStore, dataDir, agentOpts)\nif err != nil {\n    if errors.Is(err, os.ErrPermission) || strings.Contains(err.Error(), \"lock\") {\n        logger.Error(\"agent db open blocked by fs/lock\", \"dir\", dataDir, \"err\", err)\n    }\n    return errors.Wrap(err, \"start remote write agent db\")\n}","preventionTips":["Give the data-dir volume a dedicated PVC per replica.","Align fsGroup/runAsUser with volume ownership.","Monitor disk usage; clear corrupt WAL only with a backup."],"tags":["thanos","tsdb","storage","remote-write"],"backgroundTag":"file-write-permission-denied","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}