kubernetes/kubernetes · error

mount failed for NFS V3 even after running rpcBind %s, %v

Error message

mount failed for NFS V3 even after running rpcBind %s, %v

What it means

Returned by mountInChroot (mounter.go:90) when the NFS V3 mount failed with the rpc.statd message, rpcbind was then started successfully, but the retried mount still failed. The original lock issue was a symptom; the deeper NFS problem (server reachability, export permission, version mismatch) remains.

Source

Thrown at cluster/gce/gci/mounter/mounter.go:90

		return nil
	}

	if !strings.EqualFold(string(output), nfsRPCBindErrMsg) {
		// Mount failed but not because of RPC bind error
		return fmt.Errorf("mount failed: %v\nMounting command: %s\nMounting arguments: %v\nOutput: %s", err, chrootCmd, args, string(output))
	}

	// Mount failed because it is NFS V3 and we need to run rpcBind
	output, err = exec.Command(chrootCmd, rootfsPath, rpcBindCmd, "-w").CombinedOutput()
	if err != nil {
		return fmt.Errorf("mount issued for NFS V3 but unable to run rpcbind:\n Output: %s\n Error: %v", string(output), err)
	}

	// Rpcbind is running, try mounting again
	output, err = exec.Command(chrootCmd, args...).CombinedOutput()

	if err != nil {
		return fmt.Errorf("mount failed for NFS V3 even after running rpcBind %s, %v", string(output), err)
	}

	return nil
}

View on GitHub (pinned to b882c60b40)

Solutions

1. Inspect the Output field for the kernel NFS error ('access denied', 'no route to host', 'Connection refused')
2. Confirm the NFS server exports include the node IP and permit the requested NFS version
3. Check network/firewall reachability to the server on ports 2049 and 111
4. If the server is NFSv4-only, set fsType: nfs4 in the PV instead of mounting as v3

Example fix

# before: NFSv4-only server mounted as v3
apiVersion: v1
kind: PersistentVolume
spec:
  nfs:
    server: nfs.example.com
    path: /export

# after: use nfs4
spec:
  nfs:
    server: nfs.example.com
    path: /export
  # and on the node, mount -t nfs4 ...
# or open firewall for 2049/111 and confirm showmount -e nfs.example.com lists /export

// Validate NFS server reachability and export before the node attempts the mount.
func validateNFSExport(ctx context.Context, server, export string) error {
    d := net.Dialer{Timeout: 2 * time.Second}
    conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(server, "2049"))
    if err != nil {
        return fmt.Errorf("NFS server %s:2049 unreachable: %w", server, err)
    }
    conn.Close()
    return nil
}

// If the post-rpcbind retry still fails, escalate to a server-side/network diagnosis.
if err := mountInChroot(rootfs, args); err != nil {
    if strings.Contains(err.Error(), "mount failed for NFS V3 even after running rpcBind") {
        // server/firewall/export problem, not a local rpcbind problem
        return diagnoseNFS(err)
    }
    return err
}

When it happens

Trigger: NFS server unreachable or doesn't export to the node's IP; firewall blocks NFS ports (2049, mountd); server is NFSv4-only but mounted as v3; auth mismatch (sec=sys vs krb5).

Common situations: Cloud firewall/security group blocks NFS; NFS export list excludes the node subnet; typo in server address; NFSv4-only storage mounted without fsType nfs4.

Related errors

• mount issued for NFS V3 but unable to run rpcbind: Output:
• mount failed: %v Mounting command: %s Mounting arguments: %v
• path <%s> does not exist
• nodeName is empty
• could not find CSINodeLister in attachDetachController

AI-assisted analysis of kubernetes/kubernetes@b882c60b40 (2026-08-07). Data as JSON: /api/errors/32a84fcafad4e682. Report an issue: GitHub.