vitessio/vitess · error

permissions diffs: %v

Error message

permissions diffs: %v

What it means

ValidatePermissionsShard compares permissions across all tablets in a shard and returns this error when any of the parallel diffPermissions goroutines recorded a difference or failure in the shared error recorder. It is a top-level aggregation of per-tablet permission mismatches, not a single specific failure.

Source

Thrown at go/vt/wrangler/permissions.go:95

	// replicating from the primary
	aliases, err := wr.ts.FindAllTabletAliasesInShard(ctx, keyspace, shard)
	if err != nil {
		return err
	}

	// then diff all of them, except primary
	er := concurrency.AllErrorRecorder{}
	wg := sync.WaitGroup{}
	for _, alias := range aliases {
		if topoproto.TabletAliasEqual(alias, si.PrimaryAlias) {
			continue
		}
		wg.Add(1)
		go wr.diffPermissions(ctx, primaryPermissions, si.PrimaryAlias, alias, &wg, &er)
	}
	wg.Wait()
	if er.HasErrors() {
		return fmt.Errorf("permissions diffs: %v", er.Error().Error())
	}
	return nil
}

// ValidatePermissionsKeyspace validates all the permissions are the same
// in a keyspace
func (wr *Wrangler) ValidatePermissionsKeyspace(ctx context.Context, keyspace string) error {
	// find all the shards
	shards, err := wr.ts.GetShardNames(ctx, keyspace)
	if err != nil {
		return err
	}

	// corner cases
	if len(shards) == 0 {
		return fmt.Errorf("no shards in keyspace %v", keyspace)
	}
	sort.Strings(shards)

View on GitHub (pinned to 01a25a7d17)

Solutions

1. Compare the printed diffs against the primary tablet's permissions (mysql -e 'select user,host from mysql.user' etc.) on the offending tablet.
2. Re-apply the missing/extra GRANT and REVOKE statements on the divergent tablets so they match the primary.
3. If a tablet is irrecoverably drifted, re-provision or rebuild it from a fresh backup and re-apply standard grants.
4. Re-run ValidatePermissionsShard to confirm the diff is gone.

Example fix

-- before (replica missing a grant)
-- replica has no grant for app user
-- after
GRANT SELECT, INSERT, UPDATE, DELETE ON vitess.* TO 'app'@'%';

for _, tab := range tablets { if _, err := runGrantQuery(tab, "SELECT COUNT(*) FROM mysql.user"); err != nil { return err } } // also diff grants vs primary before validating

func hasPermissionDrift(primary, replica string) bool { return primary != replica }

err := wr.ValidatePermissionsShard(ctx, keyspace, shard)
if err != nil {
    if strings.Contains(err.Error(), "permissions diffs") {
        log.Warn("tablet permission drift detected; reconcile grants", slog.Any("error", err))
        return reconcileGrantsFromPrimary()
    }
    return err
}

When it happens

Trigger: Running `vtctlclient ValidatePermissionsShard <keyspace>/<shard>` when at least one tablet in the shard has permissions (MySQL grants) that differ from the shard primary's permissions.

Common situations: A replica was rebuilt or restored from a backup without reapplying GRANT statements; a new tablet was provisioned with default grants; manual grant changes applied only on the primary; per-tablet mysql.db/mysql.user tables drifted after a version upgrade.

Related errors

• validateWorkflowName.VReplicationExec: <dynamic validation.m
• invalid tablet type %v: %v
• type %v is not one of: %v
• invalid value for on-ddl: %v
• invalid action for Migrate: %s

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/ec953e2019b47753. Report an issue: GitHub.