t8y2/dbx · warning

consumer lag for group %s is incomplete: %w

Error message

consumer lag for group %s is incomplete: %w

What it means

After merging partial consume stats, examineConsumeStatsByTopic returns this error when the merged OffsetTable is empty AND successCount != len(addresses) — i.e. at least one broker succeeded but returned no offset entries, and others failed, so the lag picture is provably incomplete. It guards against reporting totalLag=0 as healthy when part of the cluster was unreadable.

Source

Thrown at agents/drivers/rocketmq/consumers.go:566

			lastErr = requestErr
			continue
		}
		partial, decodeErr := decodeConsumeStats(response.Body)
		if decodeErr != nil {
			lastErr = decodeErr
			continue
		}
		successCount++
		for key, offset := range partial.OffsetTable {
			merged.OffsetTable[key] = offset
		}
		merged.ConsumeTps += partial.ConsumeTps
	}
	if successCount == 0 {
		return nil, fmt.Errorf("query consumer lag for group %s on all masters: %w", groupID, lastErr)
	}
	if len(merged.OffsetTable) == 0 && successCount != len(addresses) {
		return nil, fmt.Errorf("consumer lag for group %s is incomplete: %w", groupID, lastErr)
	}
	return merged, nil
}

func decodeConsumeStats(body []byte) (*admin.ConsumeStats, error) {
	var stats admin.ConsumeStats
	if err := json.Unmarshal(repairRocketMQJSON(body), &stats); err != nil {
		return nil, fmt.Errorf("decode consumer stats: %w", err)
	}
	if stats.OffsetTable == nil {
		stats.OffsetTable = make(map[string]*admin.OffsetWrapper)
	}
	return &stats, nil
}

func (a *rocketMQAgent) collectSubscriptionGroupConfigs(ctx context.Context) (map[string]*subscriptionGroupConfig, error) {
	addresses, err := a.masterBrokerAddresses("")
	if err != nil {

View on GitHub (pinned to c0390bff16)

Solutions

  1. Fix the failing brokers identified via the wrapped lastErr and re-run the lag query to get complete coverage.
  2. Check that the consumer group actually has assignments on the reachable masters (consumer connections online).
  3. Verify all masters for the topic are up: `sh mqadmin brokerStatus -b <addr>` on each.
  4. If the group legitimately has no offsets (fresh group), treat zero-table as valid only when successCount == len(addresses); wait for consumers to start before monitoring lag.
  5. Add alerting on this error so partial-outage lag data is never mistaken for zero lag.

Example fix

// before: treating any lag result as complete
result, err := agent.GetConsumerLag(ctx, group, topic)
reportLag(result) // may hide partial outage
// after: surface incomplete data distinctly
result, err := agent.GetConsumerLag(ctx, group, topic)
if err != nil && strings.Contains(err.Error(), "is incomplete") {
    markLagStale(group, topic, err) // do not report totalLag=0
    return
}
reportLag(result)
Defensive patterns

Strategy: fallback

Validate before calling

// ensure full broker coverage before trusting a zero-lag reading
route, _ := adminClient.ExamineTopicRouteInfo(ctx, topic)
masters := masterAddressesFromRoute(route)
for _, a := range masters {
    if err := pingRemoting(ctx, a, 2*time.Second); err != nil {
        return fmt.Errorf("skip lag check: master %s down, data would be incomplete", a)
    }
}

Type guard

func isIncompleteLag(err error) bool {
    return err != nil && strings.Contains(err.Error(), "consumer lag for group") && strings.Contains(err.Error(), "is incomplete")
}

Try / catch

lag, err := agent.GetConsumerLag(ctx, group, topic)
switch {
case err == nil:
    reportLag(lag)
case isIncompleteLag(err):
    markMetricStale("consumer_lag", group, topic) // never emit totalLag=0
case err != nil:
    return err
}

Prevention

When it happens

Trigger: Calling getConsumerLag on a multi-master topic where some masters succeed with empty offset tables (e.g. no queues of this topic/group on them) and the rest fail, leaving merged.OffsetTable empty with successCount < len(addresses).

Common situations: Partial broker outage during a lag scrape; one master under ACL that denies the group while others respond empty; a group that hasn't consumed on any surviving master; recently rebalanced topology where queues live on brokers that are currently down.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/fc0c2847aa4043f8. Report an issue: GitHub.