{"record":{"id":"ec953e2019b47753","repo":"vitessio/vitess","slug":"permissions-diffs-v-ec953e","errorCode":null,"errorMessage":"permissions diffs: %v","messagePattern":"permissions diffs: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/wrangler/permissions.go","lineNumber":95,"sourceCode":"\t// replicating from the primary\n\taliases, err := wr.ts.FindAllTabletAliasesInShard(ctx, keyspace, shard)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// then diff all of them, except primary\n\ter := concurrency.AllErrorRecorder{}\n\twg := sync.WaitGroup{}\n\tfor _, alias := range aliases {\n\t\tif topoproto.TabletAliasEqual(alias, si.PrimaryAlias) {\n\t\t\tcontinue\n\t\t}\n\t\twg.Add(1)\n\t\tgo wr.diffPermissions(ctx, primaryPermissions, si.PrimaryAlias, alias, &wg, &er)\n\t}\n\twg.Wait()\n\tif er.HasErrors() {\n\t\treturn fmt.Errorf(\"permissions diffs: %v\", er.Error().Error())\n\t}\n\treturn nil\n}\n\n// ValidatePermissionsKeyspace validates all the permissions are the same\n// in a keyspace\nfunc (wr *Wrangler) ValidatePermissionsKeyspace(ctx context.Context, keyspace string) error {\n\t// find all the shards\n\tshards, err := wr.ts.GetShardNames(ctx, keyspace)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t// corner cases\n\tif len(shards) == 0 {\n\t\treturn fmt.Errorf(\"no shards in keyspace %v\", keyspace)\n\t}\n\tsort.Strings(shards)","sourceCodeStart":77,"sourceCodeEnd":113,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/wrangler/permissions.go#L77-L113","documentation":"ValidatePermissionsShard compares permissions across all tablets in a shard and returns this error when any of the parallel diffPermissions goroutines recorded a difference or failure in the shared error recorder. It is a top-level aggregation of per-tablet permission mismatches, not a single specific failure.","triggerScenarios":"Running `vtctlclient ValidatePermissionsShard <keyspace>/<shard>` when at least one tablet in the shard has permissions (MySQL grants) that differ from the shard primary's permissions.","commonSituations":"A replica was rebuilt or restored from a backup without reapplying GRANT statements; a new tablet was provisioned with default grants; manual grant changes applied only on the primary; per-tablet mysql.db/mysql.user tables drifted after a version upgrade.","solutions":["Compare the printed diffs against the primary tablet's permissions (mysql -e 'select user,host from mysql.user' etc.) on the offending tablet.","Re-apply the missing/extra GRANT and REVOKE statements on the divergent tablets so they match the primary.","If a tablet is irrecoverably drifted, re-provision or rebuild it from a fresh backup and re-apply standard grants.","Re-run ValidatePermissionsShard to confirm the diff is gone."],"exampleFix":"-- before (replica missing a grant)\n-- replica has no grant for app user\n-- after\nGRANT SELECT, INSERT, UPDATE, DELETE ON vitess.* TO 'app'@'%';","handlingStrategy":"try-catch","validationCode":"for _, tab := range tablets { if _, err := runGrantQuery(tab, \"SELECT COUNT(*) FROM mysql.user\"); err != nil { return err } } // also diff grants vs primary before validating","typeGuard":"func hasPermissionDrift(primary, replica string) bool { return primary != replica }","tryCatchPattern":"err := wr.ValidatePermissionsShard(ctx, keyspace, shard)\nif err != nil {\n    if strings.Contains(err.Error(), \"permissions diffs\") {\n        log.Warn(\"tablet permission drift detected; reconcile grants\", slog.Any(\"error\", err))\n        return reconcileGrantsFromPrimary()\n    }\n    return err\n}","preventionTips":["Use one source of truth (automation) to apply GRANT statements to every tablet","Re-apply grants after any backup restore or tablet rebuild","Run ValidatePermissionsShard as a periodic CI/cron check","Never hand-edit grants on a single tablet only"],"tags":["permissions","validation","vtctl","wrangler"],"backgroundTag":"permission-mismatch","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}