t8y2/dbx · error

%s: %w

Error message

%s: %w

What it means

When a mutation was attempted on all master brokers but succeeded on only some, ensureMutationCoverage builds a summary message "failed to <action> <resource> on all masters: X of Y succeeded" and wraps the last underlying error with %s: %w. It reports a partial application across the broker set, which for consumer-group config means brokers can disagree on the setting.

Source

Thrown at agents/drivers/rocketmq/consumers.go:643

	if err != nil {
		return fmt.Errorf("encode subscription group config: %w", err)
	}
	command := remoting.NewRequest(remoting.UpdateAndCreateSubscriptionGroup, nil)
	command.Body = body
	_, err = invokeRemotingWithClient(ctx, address, command)
	return err
}

func ensureMutationCoverage(action, resource string, attempted, succeeded int, lastErr error) error {
	if attempted <= 0 {
		return fmt.Errorf("no RocketMQ master brokers available to %s %s", action, resource)
	}
	if succeeded == attempted {
		return nil
	}
	message := fmt.Sprintf("failed to %s %s on all masters: %d of %d succeeded", action, resource, succeeded, attempted)
	if lastErr != nil {
		return fmt.Errorf("%s: %w", message, lastErr)
	}
	return fmt.Errorf("%s", message)
}

func (a *rocketMQAgent) enrichConsumerGroups(ctx context.Context, client *admin.Client, rows []map[string]any) {
	for _, row := range rows {
		groupID := fmt.Sprint(row["groupId"])
		connection, err := client.ExamineConsumerConnectionInfo(ctx, groupID)
		if err != nil {
			if _, ok := row["topics"]; !ok {
				row["topics"] = []string{}
			}
			continue
		}
		row["consumeType"] = valueOrDefault(connection.ConsumeType, "UNKNOWN")
		row["messageModel"] = valueOrDefault(connection.MessageModel, "CLUSTERING")
		row["memberCount"] = len(connection.ConnectionSet)
		topics := make([]string, 0, len(connection.SubscriptionTable))

View on GitHub (pinned to c0390bff16)

Solutions

1. Read the wrapped lastErr to identify which failure mode the failing broker hit
2. Re-run the mutation after the failed broker recovers to converge the cluster state (the operation is idempotent for config updates)
3. Check connectivity/ACL to each master broker individually
4. Verify all brokers run a compatible RocketMQ version supporting the config field
5. Log which brokers succeeded vs failed to detect persistent divergence

Example fix

// before
err := alterSubscriptionGroupConfig(ctx, addrs, cfg)
if err != nil { return err } // partial failure lost detail
// after
err := alterSubscriptionGroupConfig(ctx, addrs, cfg)
if err != nil {
    var partial *PartialMutationError
    if errors.As(err, &partial) && partial.Succeeded > 0 {
        // retry failed brokers only, then re-read config to verify convergence
    }
    return err
}

for _, addr := range masterAddrs {
    if !brokerHealthy(ctx, addr) {
        return fmt.Errorf("broker %s unhealthy; fix before group config mutation", addr)
    }
}

err := alterSubscriptionGroupConfig(ctx, addrs, cfg)
if err != nil {
    // partial failure: retry a bounded number of times, then re-read configs
    for i := 0; i < 3; i++ {
        if err = alterSubscriptionGroupConfig(ctx, addrs, cfg); err == nil {
            break
        }
        time.Sleep(time.Duration(1<<i) * time.Second)
    }
    if err != nil {
        log.Printf("cluster diverged after retries: %v", err)
    }
}

When it happens

Trigger: deleteConsumerGroup or alterSubscriptionGroupConfig where one or more masters fail (network timeout, broker readonly, auth rejection, version mismatch) while others succeed, so succeeded < attempted.

Common situations: One broker down or partitioned in a multi-master cluster; a lagging broker on an older RocketMQ version rejecting the update; TLS/ACL misconfigured on a single node; transient remoting timeout under load.

Related errors

• consumer lag for group %s is incomplete: %w
• %s
• query topic stats for %s on all masters: %w
• RocketMQ agent is not connected
• serialconsistency must be SERIAL or LOCAL_SERIAL

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/1ee05c496bf84991. Report an issue: GitHub.