t8y2/dbx · error

query topic stats for %s on all masters: %w

Error message

query topic stats for %s on all masters: %w

What it means

After querying topic stats on each master broker, if any master failed (successCount < len(addresses)) the function fails the whole operation, wrapping the last underlying error with this message. Partially merged results are discarded so callers never receive incomplete stats.

Source

Thrown at agents/drivers/rocketmq/topics.go:301

	for _, address := range addresses {
		response, requestErr := invokeRemotingWithClient(ctx, address,
			remoting.NewRequest(remoting.GetTopicStatsInfo, map[string]string{"topic": topic}))
		if requestErr != nil {
			lastErr = requestErr
			continue
		}
		partial, decodeErr := decodeTopicStats(response.Body)
		if decodeErr != nil {
			lastErr = decodeErr
			continue
		}
		successCount++
		for key, offset := range partial {
			merged[key] = offset
		}
	}
	if successCount != len(addresses) {
		return nil, fmt.Errorf("query topic stats for %s on all masters: %w", topic, lastErr)
	}
	if len(merged) == 0 {
		return nil, fmt.Errorf("topic stats not found: %s", topic)
	}
	return merged, nil
}

func decodeTopicStats(body []byte) (map[string]*admin.TopicOffset, error) {
	var stats admin.TopicStatsTable
	if err := json.Unmarshal(repairRocketMQJSON(body), &stats); err != nil {
		return nil, fmt.Errorf("decode topic stats: %w", err)
	}
	if stats.OffsetTable == nil {
		stats.OffsetTable = make(map[string]*admin.TopicOffset)
	}
	return stats.OffsetTable, nil
}

View on GitHub (pinned to c0390bff16)

Solutions

1. Read the wrapped lastErr to identify the failing broker and root cause (timeout vs refused vs busy)
2. Check broker health/logs for the master that failed and restart or recover it
3. Retry the query once brokers are back; transient timeouts often succeed on retry
4. If one master is persistently bad, remove it from the cluster or rebalance the topic's queues off it

Example fix

// before
stats, err := examineTopicStats(ctx, client, topic) // fails if ANY master fails
// after
stats, err := examineTopicStats(ctx, client, topic)
if err != nil {
    time.Sleep(2 * time.Second) // backoff, then retry once for transient broker errors
    stats, err = examineTopicStats(ctx, client, topic)
}

// pre-check each master is reachable before querying stats
for _, addr := range masterAddressesFromRoute(route) {
    conn, err := net.DialTimeout("tcp", addr, 2*time.Second)
    if err != nil {
        return fmt.Errorf("master %s unreachable: %w", addr, err)
    }
    conn.Close()
}

stats, err := getTopicStats(ctx, client, topic)
if err != nil && strings.Contains(err.Error(), "query topic stats") {
    if errors.Is(err, context.DeadlineExceeded) || isTransient(err) {
        // backoff and retry; transient single-master failures self-heal
    }
}

When it happens

Trigger: One or more master brokers for the topic fail invokeRemotingWithClient (connection refused, timeout, broker busy) during getTopicStats/listProducers, even if other masters succeeded.

Common situations: A master broker down or restarting during the query; network partition to one broker; broker overloaded (busy) rejecting requests; mixed broker versions where one master rejects the stats command.

Related errors

• query messages for topic %s on all masters: %w
• no RocketMQ broker address found
• query consumer status for group %s on all masters: %w
• query consumer lag for group %s on all masters: %w
• consumer lag for group %s is incomplete: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/d35a6271a4b5190a. Report an issue: GitHub.