t8y2/dbx · error

%s

Error message

%s

What it means

This error is produced by ensureMutationCoverage in the RocketMQ agent driver after a mutation (delete/alter of a consumer group or subscription group config) that must be applied to all master brokers. It reports that only some masters succeeded, wrapping the last broker error if one was captured, e.g. 'failed to delete group X on all masters: 2 of 4 succeeded: <cause>'. The library throws it because a partially-applied mutation leaves cluster state inconsistent and must not be reported as success.

Source

Thrown at agents/drivers/rocketmq/consumers.go:645

	}
	command := remoting.NewRequest(remoting.UpdateAndCreateSubscriptionGroup, nil)
	command.Body = body
	_, err = invokeRemotingWithClient(ctx, address, command)
	return err
}

func ensureMutationCoverage(action, resource string, attempted, succeeded int, lastErr error) error {
	if attempted <= 0 {
		return fmt.Errorf("no RocketMQ master brokers available to %s %s", action, resource)
	}
	if succeeded == attempted {
		return nil
	}
	message := fmt.Sprintf("failed to %s %s on all masters: %d of %d succeeded", action, resource, succeeded, attempted)
	if lastErr != nil {
		return fmt.Errorf("%s: %w", message, lastErr)
	}
	return fmt.Errorf("%s", message)
}

func (a *rocketMQAgent) enrichConsumerGroups(ctx context.Context, client *admin.Client, rows []map[string]any) {
	for _, row := range rows {
		groupID := fmt.Sprint(row["groupId"])
		connection, err := client.ExamineConsumerConnectionInfo(ctx, groupID)
		if err != nil {
			if _, ok := row["topics"]; !ok {
				row["topics"] = []string{}
			}
			continue
		}
		row["consumeType"] = valueOrDefault(connection.ConsumeType, "UNKNOWN")
		row["messageModel"] = valueOrDefault(connection.MessageModel, "CLUSTERING")
		row["memberCount"] = len(connection.ConnectionSet)
		topics := make([]string, 0, len(connection.SubscriptionTable))
		for topic := range connection.SubscriptionTable {
			topics = append(topics, topic)

View on GitHub (pinned to c0390bff16)

Solutions

1. Inspect the wrapped cause (%w) to identify which broker failed and why.
2. Check connectivity and health of every master broker (admin operation on each broker address).
3. Re-run the mutation after fixing the failing broker; operations are idempotent-ish (delete on missing group may need existence check).
4. Verify broker ACL/credentials are identical across all masters.
5. If the resource is intentionally missing on some brokers, align cluster state or scope the operation per-broker.

Example fix

// before
err := agent.DeleteConsumerGroup(ctx, map[string]any{"groupName": "GID_demo"})
// after (check partial-apply cause and retry per failing broker)
if err != nil {
    var partial *PartialMutationError // or unwrap the wrapped lastErr
    if errors.As(err, &partial) {
        log.Printf("applied to %d/%d masters, cause: %v", partial.Succeeded, partial.Attempted, partial.LastErr)
        // fix/retry the failing broker before treating the delete as done
    }
}

// Check all masters are reachable before mutating
for _, addr := range masterAddrs {
    if err := probeBroker(ctx, addr); err != nil {
        return fmt.Errorf("master %s unreachable, aborting mutation: %w", addr, err)
    }
}

err := agent.DeleteConsumerGroup(ctx, params)
if err != nil {
    var cause error
    if errors.As(err, &target) || errors.Unwrap(err) != nil {
        cause = errors.Unwrap(err)
    }
    log.Printf("partial mutation: %v (root cause: %v)", err, cause)
    // verify state per broker, then retry only the failed masters
}

When it happens

Trigger: Calling deleteConsumerGroup or alterSubscriptionGroupConfig in a multi-master cluster where one or more brokers reject or fail the request (broker down, group/config missing on one broker, permission denied, timeout on one master).

Common situations: Cluster with unequal state after a broker was replaced; ACL/permission differences between brokers; one master restarted and out of sync; network partition to a single broker while others respond fine.

Related errors

• consumer lag for group %s is incomplete: %w
• %s: %w
• query topic stats for %s on all masters: %w
• RocketMQ agent is not connected
• decode ACL list: %w

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/042c8375688f2430. Report an issue: GitHub.