{"record":{"id":"e003ea844db3f5eb","repo":"thanos-io/thanos","slug":"s-command-failed","errorCode":null,"errorMessage":"%s command failed","messagePattern":"(.+?) command failed","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/main.go","lineNumber":169,"sourceCode":"\t\t\treturn interrupt(logger, cancel)\n\t\t}, func(error) {\n\t\t\tclose(cancel)\n\t\t})\n\t}\n\n\t// Listen for reload signals.\n\t{\n\t\tcancel := make(chan struct{})\n\t\tg.Add(func() error {\n\t\t\treturn reload(logger, cancel, reloadCh)\n\t\t}, func(error) {\n\t\t\tclose(cancel)\n\t\t})\n\t}\n\n\tif err := g.Run(); err != nil {\n\t\t// Use %+v for github.com/pkg/errors error to print with stack.\n\t\tlevel.Error(logger).Log(\"err\", fmt.Sprintf(\"%+v\", errors.Wrapf(err, \"%s command failed\", cmd)))\n\t\tos.Exit(1)\n\t}\n\tlevel.Info(logger).Log(\"msg\", \"exiting\")\n}\n\nfunc interrupt(logger log.Logger, cancel <-chan struct{}) error {\n\tc := make(chan os.Signal, 1)\n\tsignal.Notify(c, syscall.SIGINT, syscall.SIGTERM)\n\tselect {\n\tcase s := <-c:\n\t\tlevel.Info(logger).Log(\"msg\", \"caught signal. Exiting.\", \"signal\", s)\n\t\treturn nil\n\tcase <-cancel:\n\t\treturn errors.New(\"canceled\")\n\t}\n}\n\nfunc reload(logger log.Logger, cancel <-chan struct{}, r chan<- struct{}) error {","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/main.go#L151-L187","documentation":"Top-level wrapper for the run group: after setup succeeds, Thanos runs all actors via g.Run(); if any actor in the group returns an error, main wraps it with \"%s command failed\" and logs it with a stack trace, then exits 1. This signals a runtime failure of the command while it was running, not a setup failure.","triggerScenarios":"Any actor registered in the run group fails at runtime: gRPC/HTTP listener bind errors, store endpoint connection failures, TLS handshake setup errors, interrupted/canceled context, or component loops returning errors.","commonSituations":"Port already in use, TLS certificate files unreadable at runtime, downstream store APIs unavailable, or the process received a cancel trigger from another failing actor.","solutions":["Inspect the full %+v stack trace to identify which run-group actor failed.","Check that listen addresses (--grpc-address, --http-address) are free: `ss -ltnp` or `lsof -i :<port>`.","Verify connectivity to configured store/endpoint peers.","Check TLS cert/key file existence and permissions if TLS is enabled.","Restart the command after fixing the root cause."],"exampleFix":"// before\nthanos query --grpc-address=0.0.0.0:9090  // port already bound\n// after\nthanos query --grpc-address=0.0.0.0:9091","handlingStrategy":"retry","validationCode":"# Check listen ports are free before start\nss -ltn | grep -E ':(9090|10902)' || echo \"ports free\"","typeGuard":null,"tryCatchPattern":"// systemd unit\nRestart=on-failure\nRestartSec=5s","preventionTips":["Reserve ports for gRPC/HTTP and verify with a pre-start check","Mount TLS secrets before process start and verify with an initContainer","Monitor the specific wrapped inner error, not the generic command-failed wrapper"],"tags":["cli","runtime","run-group"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}