What causes "Invalid public key" in rustdesk/rustdesk-server?

Re-copy the public key exactly as printed by the key generator (single line of base64).

Invalid public key

Error message

Invalid public key

What it means

validate_keypair base64-decodes the supplied public key; if `base64::decode(pk)` returns Err, it bails with "Invalid public key". As with the secret-key check, this fires before any crypto: the -p/public-key argument is not valid base64.

Solutions

Re-copy the public key exactly as printed by the key generator (single line of base64).
Single-quote the argument in the shell to protect '+', '/' and '=' characters.
Verify decodability: `echo '<pk>' | base64 -d > /dev/null` must succeed.
If the .pub file contains extra headers/wrappers, extract only the base64 payload.

Example fix

// before (newline-broken paste)
-K 'abcCdef
+ghi='
// after (single-quoted, one line)
-K 'abcCdef+ghi='

Defensive patterns

Strategy: validation

Validate before calling

fn is_base64(s: &str) -> bool { base64::decode(s.trim()).is_ok() }
// pre-check: assert!(is_base64(&pk), "public key must be valid base64");

Type guard

fn valid_pk_base64(s: &str) -> Option<Vec<u8>> { base64::decode(s.trim()).ok().filter(|b| b.len() == 32) }

Try / catch

match validate_keypair(&pk, &sk) {
    Err(e) if e.to_string().contains("public key") => eprintln!("Public key must be one line of base64"),
    Err(e) => eprintln!("keypair check failed: {e}"),
    Ok(()) => println!("keypair OK"),
}

Prevention

Extract only the base64 payload if the .pub file has wrappers.
Strip trailing newlines when reading keys from files (e.g. tr -d '\n').
Single-quote key arguments in the shell.

When it happens

Trigger: Passing a public key argument containing invalid base64 characters, bad padding, whitespace/newlines, or an empty string so base64::decode fails.

Common situations: Copy-paste errors from the id_ed25519.pub output; shell interpolation inserting extra characters; confusing the server's public key file with an HTML/PEM wrapper around it.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of rustdesk/rustdesk-server@a7736be5e4 (2026-09-09). Data as JSON: /api/errors/44c7a9a23fe0cb95. Report an issue: GitHub.

Appendix: source

Thrown at src/utils.rs:50

    println!("Secret Key:  {secret_key}");
}

fn validate_keypair(pk: &str, sk: &str) -> ResultType<()> {
    let sk1 = base64::decode(sk);
    if sk1.is_err() {
        bail!("Invalid secret key");
    }
    let sk1 = sk1.unwrap();

    let secret_key = sign::SecretKey::from_slice(sk1.as_slice());
    if secret_key.is_none() {
        bail!("Invalid Secret key");
    }
    let secret_key = secret_key.unwrap();

    let pk1 = base64::decode(pk);
    if pk1.is_err() {
        bail!("Invalid public key");
    }
    let pk1 = pk1.unwrap();

    let public_key = sign::PublicKey::from_slice(pk1.as_slice());
    if public_key.is_none() {
        bail!("Invalid Public key");
    }
    let public_key = public_key.unwrap();

    let random_data_to_test = b"This is meh.";
    let signed_data = sign::sign(random_data_to_test, &secret_key);
    let verified_data = sign::verify(&signed_data, &public_key);
    if verified_data.is_err() {
        bail!("Key pair is INVALID");
    }
    let verified_data = verified_data.unwrap();

    if random_data_to_test != &verified_data[..] {

View on GitHub (pinned to a7736be5e4)