weaviate/weaviate · error

restore roles: %w

Error message

restore roles: %w

What it means

ValidateBackupSnapshot verifies a backup's roles before a cluster-wide restore. When namespaces are enabled, RequireReferencedNamespacesExist checks that every namespace referenced by the backup's roles exists and is not being deleted; a failure is wrapped as "restore roles: %w". This is a pre-flight safety check so restoring roles never references namespaces that don't exist on this cluster.

Source

Thrown at cluster/rbac/manager.go:322

// If this cluster uses namespaces, every namespace the roles name must exist
// and not be deleting. Suspended and resuming namespaces are accepted because
// they keep their rows, so restoring those rows is legal and must not block a
// cluster-wide restore.
func (m *Manager) ValidateBackupSnapshot(req *cmd.RestoreRolesAndUsersRequest, ns usecasesNamespaces.Exister) error {
	if m.authZ == nil || len(req.Roles) == 0 {
		return nil
	}
	staticAPIKeyUsers := rbac.StaticAPIKeyUsers(m.authNconfig)
	if err := rbac.ValidateSnapshot(req.Roles, req.StripNamespaces, staticAPIKeyUsers); err != nil {
		return err
	}
	if req.StripNamespaces {
		// This cluster has namespaces turned off, so there is no namespace here
		// that could be active. The check above covers this case instead.
		return nil
	}
	if err := rbac.RequireReferencedNamespacesExist(req.Roles, staticAPIKeyUsers, ns); err != nil {
		return fmt.Errorf("restore roles: %w", err)
	}
	return nil
}

// RestoreFromBackup replaces every role with the ones from the backup.
// Not to be confused with Restore, which loads roles when a node starts up.
func (m *Manager) RestoreFromBackup(req *cmd.RestoreRolesAndUsersRequest) error {
	if m.authZ == nil || len(req.Roles) == 0 {
		return nil
	}
	if err := m.authZ.Restore(req.Roles, req.StripNamespaces); err != nil {
		// The restore wipes the old roles before the part that can fail, so this
		// node may now have no custom roles at all while every other node
		// succeeded. Log a fixed word so this is easy to search for.
		m.logger.WithField("action", "restore_roles_from_backup").
			Errorf("rbac_restore_torn: role store may be cleared on this node only: %v", err)
		return err
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Recreate (or stop deleting) the namespaces referenced by the backup's roles on the target cluster, then retry the restore
2. Inspect the backup's role list to find which namespaces are referenced and compare with the target cluster
3. Re-take the backup after aligning namespace configuration between source and target clusters

// before restoring: confirm every namespace referenced by backup roles exists and is not deleting
for _, ns := range namespacesReferencedByBackupRoles(backup.Roles) {
  if err := nsExister.Exists(ns); err != nil { return fmt.Errorf("namespace %s unavailable for restore", ns) }
}

if err := manager.ValidateBackupSnapshot(req, nsExister); err != nil {
  if strings.HasPrefix(err.Error(), "restore roles:") { /* reconcile namespaces then retry */ }
}

When it happens

Trigger: Restoring a backup whose roles reference namespaces that (a) don't exist on the target cluster, or (b) exist but are in the DELETING state. Suspended/resuming namespaces are accepted; active-deletion is not.

Common situations: Restoring a backup from cluster A into cluster B with a different namespace layout; attempting a restore while someone is concurrently deleting a namespace; operator disabled/renamed namespaces between backup and restore.

Related errors

• backup references namespaces that are missing or deleting on
• no roles or users to restore: %w
• unmarshal restore-roles-and-users subcommand: %w
• 'includeRoles' was set but RBAC is not enabled
• a namespace-local role can only be assigned by an administra

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/0e003f8ef01382ec. Report an issue: GitHub.