What causes "query consumer lag for group %s on all masters: %w" in t8y2/dbx?

Inspect the wrapped cause (errors.Unwrap) for the real per-broker failure and address it (timeout, auth, decode).

query consumer lag for group %s on all masters: %w

Error message

query consumer lag for group %s on all masters: %w

What it means

After querying GetConsumeStats on every master broker, examineConsumeStatsByTopic returns this error when successCount == 0 — no master returned a decodable consume-stats response. The wrapped lastErr carries the final underlying failure (request error or decode error) from the last attempted broker.

Source

Thrown at agents/drivers/rocketmq/consumers.go:563

			map[string]string{"consumerGroup": groupID, "topic": topic},
		))
		if requestErr != nil {
			lastErr = requestErr
			continue
		}
		partial, decodeErr := decodeConsumeStats(response.Body)
		if decodeErr != nil {
			lastErr = decodeErr
			continue
		}
		successCount++
		for key, offset := range partial.OffsetTable {
			merged.OffsetTable[key] = offset
		}
		merged.ConsumeTps += partial.ConsumeTps
	}
	if successCount == 0 {
		return nil, fmt.Errorf("query consumer lag for group %s on all masters: %w", groupID, lastErr)
	}
	if len(merged.OffsetTable) == 0 && successCount != len(addresses) {
		return nil, fmt.Errorf("consumer lag for group %s is incomplete: %w", groupID, lastErr)
	}
	return merged, nil
}

func decodeConsumeStats(body []byte) (*admin.ConsumeStats, error) {
	var stats admin.ConsumeStats
	if err := json.Unmarshal(repairRocketMQJSON(body), &stats); err != nil {
		return nil, fmt.Errorf("decode consumer stats: %w", err)
	}
	if stats.OffsetTable == nil {
		stats.OffsetTable = make(map[string]*admin.OffsetWrapper)
	}
	return &stats, nil
}

View on GitHub (pinned to c0390bff16)

Solutions

Inspect the wrapped cause (errors.Unwrap) for the real per-broker failure and address it (timeout, auth, decode).
Verify network reachability to each master's remoting port from the agent host.
Confirm the consumer group exists and has valid subscriptions (a never-started group may be rejected); check ACL settings if enabled.
Check broker logs for errors correlated with the GetConsumeStats request timestamp.
Retry after broker load/health issues subside; a single healthy master makes successCount >= 1.

Example fix

// before: fail outright when every master fails
stats, err := agent.GetConsumerLag(ctx, group, topic)
// after: log the wrapped cause and retry transient failures
stats, err := agent.GetConsumerLag(ctx, group, topic)
if err != nil && isTransient(errors.Unwrap(err)) {
    stats, err = retryWithBackoff(3, func() error {
        var e error
        stats, e = agent.GetConsumerLag(ctx, group, topic)
        return e
    })
}

Defensive patterns

Strategy: retry

Validate before calling

// pre-flight: confirm at least one master answers before lag collection
for _, a := range masterAddresses {
    if err := pingRemoting(ctx, a, 2*time.Second); err != nil {
        log.Printf("master %s not answering: %v", a, err)
    }
}

Type guard

func isLagQueryAllFailed(err error) bool {
    return err != nil && strings.Contains(err.Error(), "query consumer lag") && strings.Contains(err.Error(), "on all masters")
}

Try / catch

lag, err := agent.GetConsumerLag(ctx, group, topic)
if err != nil && isLagQueryAllFailed(err) {
    if backoffErr := retryWithBackoff(3, time.Second, func() error {
        var e error
        lag, e = agent.GetConsumerLag(ctx, group, topic)
        return e
    }); backoffErr != nil {
        return fmt.Errorf("lag unavailable for %s, cause=%v", group, errors.Unwrap(backoffErr))
    }
}

Prevention

Open firewall paths to broker remoting port (10911) from the agent host.
Verify consumer group existence and ACL permissions before scraping lag.
Alert on broker CPU/connection saturation that causes request timeouts.
Keep broker versions aligned to avoid decode failures counted as request failures.

When it happens

Trigger: Calling getConsumerLag when all masters fail the GetConsumeStats remoting call: network timeouts, brokers rejecting the consumerGroup/topic request, ACL authorization failures, or decodeConsumeStats failing on every body.

Common situations: Firewall blocking broker port 10911 from the agent host; consumer group doesn't exist so brokers return errors; ACL credentials missing or expired; brokers overloaded and timing out; mixed broker versions emitting bodies the JSON repairer can't parse.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/cd22ce552f452051. Report an issue: GitHub.