clockworklabs/SpacetimeDB · critical · anyhow::Error

Unable to read private key for JWT token signing

Error message

Unable to read private key for JWT token signing

What it means

On startup the node loads the EC key pair used to sign JWTs. If the public key file loads but the private key file cannot be read (missing, unreadable, or corrupt), startup aborts with this message. The node will not regenerate just one half of the pair -- keys are only auto-created when both files are absent, so a missing private key must be restored or the whole pair regenerated.

Source

Thrown at crates/core/src/auth/mod.rs:66

// Get the key pair if the given files exist. If they don't, create them.
// If only one of the files exists, return an error.
pub fn get_or_create_keys(certs: &CertificateAuthority) -> anyhow::Result<JwtKeys> {
    let public_key_path = &certs.jwt_pub_key_path;
    let private_key_path = &certs.jwt_priv_key_path;

    let public_key_bytes = public_key_path.read().ok();
    let private_key_bytes = private_key_path.read().ok();

    // If both keys are unspecified, create them
    let key_pair = match (public_key_bytes, private_key_bytes) {
        (Some(pub_), Some(priv_)) => EcKeyPair::new(pub_, priv_),
        (None, None) => {
            let keys = EcKeyPair::generate()?;
            keys.write_to_files(public_key_path, private_key_path)?;
            keys
        }
        (None, Some(_)) => anyhow::bail!("Unable to read public key for JWT token verification"),
        (Some(_), None) => anyhow::bail!("Unable to read private key for JWT token signing"),
    };

    key_pair.try_into()
}

// An Ec key pair in pem format.
pub struct EcKeyPair {
    pub public_key_bytes: Vec<u8>,
    pub private_key_bytes: Vec<u8>,
}

impl TryFrom<EcKeyPair> for JwtKeys {
    type Error = anyhow::Error;
    fn try_from(pair: EcKeyPair) -> anyhow::Result<Self> {
        JwtKeys::new(pair.public_key_bytes, pair.private_key_bytes)
    }
}

View on GitHub (pinned to 6dee26c6ef)

Solutions

1. Restore the matching private key from your secret backup (it must correspond to the existing public key)
2. Otherwise regenerate the pair: delete BOTH key files and restart the node -- a fresh pair is generated and persisted (all previously issued JWTs become invalid)
3. Fix permissions/ownership so the node process can read the private key
4. Verify the pair matches: `openssl pkey -in private.pem -pubout` vs the public key file

Example fix

# before: only the public key is present
$ ls /keys
jwt_public.pem

# after: restore the matching private key, or regenerate the whole pair
$ cp /backup/jwt_private.pem /keys/ && chown spacetimedb /keys/jwt_private.pem && systemctl restart spacetimedb

#!/usr/bin/env bash
# run before starting the node
for f in "$JWT_PUBLIC_KEY" "$JWT_PRIVATE_KEY"; do
  [ -r "$f" ] && [ -s "$f" ] || { echo "missing/empty key: $f" >&2; exit 1; }
done
openssl pkey -in "$JWT_PRIVATE_KEY" -pubout 2>/dev/null | diff -q - "$JWT_PUBLIC_KEY" \
  || echo 'WARNING: key pair mismatch' >&2

When it happens

Trigger: The public key file exists but the private key file was deleted, moved, has a 0-byte/truncated body, or permissions deny the node process read access; a backup that captured only the public half; key paths misconfigured so they point at different directories.

Common situations: Secrets scrubbing (e.g. deploy pipelines or .gitignore rules) removing the private key but leaving the public one; a failed key rotation; Docker secrets mounted incorrectly; running under a different user than the key files' owner.

Related errors

• Unable to read public key for JWT token verification
• Invalid JSON: failed to parse string
• Token does not look like a JSON web token: {token}
• Issuer too long: {:?}
• Subject too long: {:?}

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/46689f5056b3676d. Report an issue: GitHub.