rqlite/rqlite · warning

remote remove node not authorized

Error message

remote remove node not authorized

What it means

In http.Service.handleRemove, when the leader-forwarded remove-node call returns proxy.ErrUnauthorized, the handler responds HTTP 401 with the literal body 'remote remove node not authorized'. This means the remote leader rejected the node-remove request because the client's basic-auth credentials are missing or lack permission for the operation. It is an authentication/authorization failure, not a transport or cluster-state problem.

Source

Thrown at http/service.go:577

	}

	rn := &proto.RemoveNodeRequest{
		Id: remoteID,
	}

	addr, err := s.proxy.Remove(r.Context(), rn, makeCredentials(r), qp.Timeout(defaultTimeout), qp.Redirect())
	if err != nil {
		if errors.Is(err, proxy.ErrNotLeader) {
			s.DoRedirect(w, r, qp)
			return
		}
		if errors.Is(err, proxy.ErrLeaderNotFound) {
			stats.Add(numLeaderNotFound, 1)
			http.Error(w, proxy.ErrLeaderNotFound.Error(), http.StatusServiceUnavailable)
			return
		}
		if errors.Is(err, proxy.ErrUnauthorized) {
			http.Error(w, "remote remove node not authorized", http.StatusUnauthorized)
			return
		}
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	w.Header().Set(ServedByHTTPHeader, addr)
}

// handleSQLAnalyze handles requests to analyze and show SQL rewriting.
func (s *Service) handleSQLAnalyze(w http.ResponseWriter, r *http.Request, qp QueryParams) {
	w.Header().Set("Content-Type", "application/json; charset=utf-8")

	if !s.CheckRequestPerm(r, auth.PermQuery) {
		w.WriteHeader(http.StatusUnauthorized)
		return
	}

	if r.Method != "GET" && r.Method != "POST" {

View on GitHub (pinned to 7586a4d1bd)

Solutions

  1. Send valid basic-auth credentials of a user with sufficient permissions: curl -XDELETE -u 'user:pass' ...
  2. Check the node's auth configuration / create or elevate a user permitted to remove nodes.
  3. Verify no intermediate proxy strips the Authorization header.
  4. Re-test; if 401 persists, confirm the credentials against the auth config on the leader node.

Example fix

// before
curl -XDELETE 'localhost:4001/db/node?addr=1.2.3.4:4002'  # 401 remote remove node not authorized
// after
curl -XDELETE -u 'admin:secret' 'localhost:4001/db/node?addr=1.2.3.4:4002'
Defensive patterns

Strategy: validation

Validate before calling

function assertRemoveAuth(creds) {
  if (!creds || !creds.username || !creds.password) {
    throw new Error('rqlite auth enabled: basic-auth credentials required for node removal')
  }
}

Try / catch

const res = await fetch(url, {method:'DELETE', headers:{Authorization: basicAuth(user, pass)}})
if (res.status === 401 && (await res.text()).includes('not authorized')) {
  // refresh credentials / verify user permission level, then retry
  throw new Error('node removal rejected: invalid or insufficient credentials')
}

Prevention

When it happens

Trigger: DELETE /db/node on an auth-enabled cluster without credentials, with wrong username/password, or with a user lacking sufficient permission level for node removal (join/remove operations require elevated auth).

Common situations: Cluster deployed with -auth but automation scripts not updated to send credentials; auth config changed (users recreated) leaving old credentials invalid; reverse proxy stripping the Authorization header; users created with too-low permission level.

Related errors

AI-assisted analysis of rqlite/rqlite@7586a4d1bd (2026-09-03). Data as JSON: /api/errors/613fc72b95808a1a. Report an issue: GitHub.