t8y2/dbx · error

query topic stats for

Error message

query topic stats for %s on all masters: %w

What it means

After querying topic stats on each master broker, if any master failed (successCount < len(addresses)) the function fails the whole operation, wrapping the last underlying error with this message. Partially merged results are discarded so callers never receive incomplete stats.

Solutions

1. Read the wrapped lastErr to identify the failing broker and root cause (timeout vs refused vs busy)
2. Check broker health/logs for the master that failed and restart or recover it
3. Retry the query once brokers are back; transient timeouts often succeed on retry
4. If one master is persistently bad, remove it from the cluster or rebalance the topic's queues off it

Example fix

// before
stats, err := examineTopicStats(ctx, client, topic) // fails if ANY master fails
// after
stats, err := examineTopicStats(ctx, client, topic)
if err != nil {
    time.Sleep(2 * time.Second) // backoff, then retry once for transient broker errors
    stats, err = examineTopicStats(ctx, client, topic)
}

// pre-check each master is reachable before querying stats
for _, addr := range masterAddressesFromRoute(route) {
    conn, err := net.DialTimeout("tcp", addr, 2*time.Second)
    if err != nil {
        return fmt.Errorf("master %s unreachable: %w", addr, err)
    }
    conn.Close()
}

stats, err := getTopicStats(ctx, client, topic)
if err != nil && strings.Contains(err.Error(), "query topic stats") {
    if errors.Is(err, context.DeadlineExceeded) || isTransient(err) {
        // backoff and retry; transient single-master failures self-heal
    }
}

When it happens

Trigger: One or more master brokers for the topic fail invokeRemotingWithClient (connection refused, timeout, broker busy) during getTopicStats/listProducers, even if other masters succeeded.

Common situations: A master broker down or restarting during the query; network partition to one broker; broker overloaded (busy) rejecting requests; mixed broker versions where one master rejects the stats command.

Related errors

• query messages for topic
• consumer lag for group
• invalid queue offset
• invalid RocketMQ frame length
• no RocketMQ broker address found

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/d35a6271a4b5190a. Report an issue: GitHub.

Appendix: source

Thrown at agents/drivers/rocketmq/topics.go:301

	for _, address := range addresses {
		response, requestErr := invokeRemotingWithClient(ctx, address,
			remoting.NewRequest(remoting.GetTopicStatsInfo, map[string]string{"topic": topic}))
		if requestErr != nil {
			lastErr = requestErr
			continue
		}
		partial, decodeErr := decodeTopicStats(response.Body)
		if decodeErr != nil {
			lastErr = decodeErr
			continue
		}
		successCount++
		for key, offset := range partial {
			merged[key] = offset
		}
	}
	if successCount != len(addresses) {
		return nil, fmt.Errorf("query topic stats for %s on all masters: %w", topic, lastErr)
	}
	if len(merged) == 0 {
		return nil, fmt.Errorf("topic stats not found: %s", topic)
	}
	return merged, nil
}

func decodeTopicStats(body []byte) (map[string]*admin.TopicOffset, error) {
	var stats admin.TopicStatsTable
	if err := json.Unmarshal(repairRocketMQJSON(body), &stats); err != nil {
		return nil, fmt.Errorf("decode topic stats: %w", err)
	}
	if stats.OffsetTable == nil {
		stats.OffsetTable = make(map[string]*admin.TopicOffset)
	}
	return stats.OffsetTable, nil
}

View on GitHub (pinned to c0390bff16)