{"record":{"id":"d35a6271a4b5190a","repo":"t8y2/dbx","slug":"query-topic-stats-for-s-on-all-masters-w","errorCode":null,"errorMessage":"query topic stats for %s on all masters: %w","messagePattern":"query topic stats for (.+?) on all masters: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"agents/drivers/rocketmq/topics.go","lineNumber":301,"sourceCode":"\tfor _, address := range addresses {\n\t\tresponse, requestErr := invokeRemotingWithClient(ctx, address,\n\t\t\tremoting.NewRequest(remoting.GetTopicStatsInfo, map[string]string{\"topic\": topic}))\n\t\tif requestErr != nil {\n\t\t\tlastErr = requestErr\n\t\t\tcontinue\n\t\t}\n\t\tpartial, decodeErr := decodeTopicStats(response.Body)\n\t\tif decodeErr != nil {\n\t\t\tlastErr = decodeErr\n\t\t\tcontinue\n\t\t}\n\t\tsuccessCount++\n\t\tfor key, offset := range partial {\n\t\t\tmerged[key] = offset\n\t\t}\n\t}\n\tif successCount != len(addresses) {\n\t\treturn nil, fmt.Errorf(\"query topic stats for %s on all masters: %w\", topic, lastErr)\n\t}\n\tif len(merged) == 0 {\n\t\treturn nil, fmt.Errorf(\"topic stats not found: %s\", topic)\n\t}\n\treturn merged, nil\n}\n\nfunc decodeTopicStats(body []byte) (map[string]*admin.TopicOffset, error) {\n\tvar stats admin.TopicStatsTable\n\tif err := json.Unmarshal(repairRocketMQJSON(body), &stats); err != nil {\n\t\treturn nil, fmt.Errorf(\"decode topic stats: %w\", err)\n\t}\n\tif stats.OffsetTable == nil {\n\t\tstats.OffsetTable = make(map[string]*admin.TopicOffset)\n\t}\n\treturn stats.OffsetTable, nil\n}\n","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/agents/drivers/rocketmq/topics.go#L283-L319","documentation":"After querying topic stats on each master broker, if any master failed (successCount < len(addresses)) the function fails the whole operation, wrapping the last underlying error with this message. Partially merged results are discarded so callers never receive incomplete stats.","triggerScenarios":"One or more master brokers for the topic fail invokeRemotingWithClient (connection refused, timeout, broker busy) during getTopicStats/listProducers, even if other masters succeeded.","commonSituations":"A master broker down or restarting during the query; network partition to one broker; broker overloaded (busy) rejecting requests; mixed broker versions where one master rejects the stats command.","solutions":["Read the wrapped lastErr to identify the failing broker and root cause (timeout vs refused vs busy)","Check broker health/logs for the master that failed and restart or recover it","Retry the query once brokers are back; transient timeouts often succeed on retry","If one master is persistently bad, remove it from the cluster or rebalance the topic's queues off it"],"exampleFix":"// before\nstats, err := examineTopicStats(ctx, client, topic) // fails if ANY master fails\n// after\nstats, err := examineTopicStats(ctx, client, topic)\nif err != nil {\n    time.Sleep(2 * time.Second) // backoff, then retry once for transient broker errors\n    stats, err = examineTopicStats(ctx, client, topic)\n}","handlingStrategy":"retry","validationCode":"// pre-check each master is reachable before querying stats\nfor _, addr := range masterAddressesFromRoute(route) {\n    conn, err := net.DialTimeout(\"tcp\", addr, 2*time.Second)\n    if err != nil {\n        return fmt.Errorf(\"master %s unreachable: %w\", addr, err)\n    }\n    conn.Close()\n}","typeGuard":null,"tryCatchPattern":"stats, err := getTopicStats(ctx, client, topic)\nif err != nil && strings.Contains(err.Error(), \"query topic stats\") {\n    if errors.Is(err, context.DeadlineExceeded) || isTransient(err) {\n        // backoff and retry; transient single-master failures self-heal\n    }\n}","preventionTips":["Set generous per-call timeouts so one slow master doesn't fail the batch","Monitor broker health and alert before querying stats","Retry transient errors with backoff","Keep cluster brokers on compatible versions to avoid per-master rejections"],"tags":["rocketmq","network","broker","partial-failure"],"backgroundTag":"broker-query-failed","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}