What causes "Custom engines not supported by this OpenSSL" in denoland/deno?

Export the key from the engine/HSM into a PEM file and pass it via options.key (with options.passphrase if encrypted) instead of privateKeyEngine/privateKeyIdentifier

denoland/deno · error · Error

ERR_CRYPTO_CUSTOM_ENGINE_NOT_SUPPORTED

ERR_CRYPTO_CUSTOM_ENGINE_NOT_SUPPORTED

Error message

Custom engines not supported by this OpenSSL

What it means

Deno's node:tls SecureContext polyfill cannot load private keys through an external OpenSSL engine, because Deno uses rustls instead of OpenSSL. Node allows pairing options.privateKeyEngine (an OpenSSL engine identifier) with options.privateKeyIdentifier (a key id like 'pkcs11:...') to reference a key held in an HSM or smartcard. This polyfill (ext/node/polyfills/_tls_common.ts:544) rejects that combination up front with ERR_CRYPTO_CUSTOM_ENGINE_NOT_SUPPORTED.

Source

Thrown at ext/node/polyfills/_tls_common.ts:544

      err.function = "dlfcn_load";
      err.reason = "could not load the shared library";
      err.code = "ERR_OSSL_DSO_COULD_NOT_LOAD_THE_SHARED_LIBRARY";
      throw err;
    }
    if (options.privateKeyEngine != null) {
      validateString(options.privateKeyEngine, "options.privateKeyEngine");
    }
    if (options.privateKeyIdentifier != null) {
      validateString(
        options.privateKeyIdentifier,
        "options.privateKeyIdentifier",
      );
    }
    if (
      options.privateKeyEngine != null &&
      options.privateKeyIdentifier != null
    ) {
      throw new ERR_CRYPTO_CUSTOM_ENGINE_NOT_SUPPORTED();
    }
    if (options.ecdhCurve != null) {
      validateString(options.ecdhCurve, "options.ecdhCurve");
    }
    // Validate cert before key - Node.js processes cert first (SetCert before SetKey)
    validateKeyCertOption(options.cert, "options.cert", false);
    validateKeyCertOption(options.key, "options.key", true);
    validateKeyCertOption(options.ca, "options.ca", false);

    // Load PFX / PKCS#12 data: extract the cert + private key so they can
    // be used by the underlying TLS implementation. Any additional certs
    // present in the PFX are merged into `ca`. Caller-supplied `cert`/`key`
    // (and `ca`) take precedence, matching Node, which loads PFX first and
    // then layers explicit cert/key on top.
    //
    // Node accepts both a single <string>|<Buffer> and an
    // <Array<string|Buffer|{ buf, passphrase? }>>; an empty array (which
    // playwright passes when no PFX is configured) must be a no-op rather

View on GitHub (pinned to 89f33cbef2)

Solutions

Export the key from the engine/HSM into a PEM file and pass it via options.key (with options.passphrase if encrypted) instead of privateKeyEngine/privateKeyIdentifier
If the key cannot leave the HSM, run that service under Node.js rather than Deno, since engine-backed keys are architecturally unsupported under rustls
Gate engine-based TLS setup behind a runtime check (e.g. process.versions.nsx or a feature flag) so Deno deployments take a non-engine code path

Example fix

// before
const ctx = tls.createSecureContext({
  privateKeyEngine: 'pkcs11',
  privateKeyIdentifier: 'pkcs11:token=mytok;object=mykey',
});

// after
const ctx = tls.createSecureContext({
  key: fs.readFileSync('/etc/tls/server.key.pem'),
  passphrase: process.env.KEY_PASSPHRASE,
});

Defensive patterns

Strategy: type-guard

Validate before calling

function usesCustomEngine(options) {
  return options?.privateKeyEngine != null && options?.privateKeyIdentifier != null;
}
if (usesCustomEngine(tlsOptions)) {
  throw new Error('engine-backed keys unsupported in this runtime; provide options.key');
}

Type guard

function hasEngineKey(o) {
  return typeof o === 'object' && o !== null &&
    o.privateKeyEngine != null && o.privateKeyIdentifier != null;
}

Try / catch

try { tls.createSecureContext(opts); } catch (e) { if (e.code === 'ERR_CRYPTO_CUSTOM_ENGINE_NOT_SUPPORTED') { /* fall back to PEM key path */ } else throw e; }

Prevention

Keep PEM key paths in runtime config; treat engine identifiers as Node-only deployment detail
Assert at startup that tls key options contain either key or a PEM path, never engine+identifier pairs

When it happens

Trigger: Calling tls.createSecureContext({ privateKeyEngine: 'pkcs11', privateKeyIdentifier: 'pkcs11:...' }), or constructing a tls.Server / https server whose options object carries both privateKeyEngine and privateKeyIdentifier (both must be non-null for the throw to fire).

Common situations: Running Node code that stores TLS keys on an HSM, YubiKey, or PKCS#11 token; enterprise apps using engine-based key loading; also triggered indirectly by packages like node-keytar or pkcs11js wrapping TLS setup. Migrating such services to Deno without changing key sourcing.

Related errors

AI-assisted analysis of denoland/deno@89f33cbef2 (2026-08-16). Data as JSON: /api/errors/000de8ede495d963. Report an issue: GitHub.