huggingface/tokenizers · critical · Error

Cannot find native binding. npm has a bug related to…

Error message

Cannot find native binding. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.

What it means

This is the loader's final fallback failure: no platform native binding could be loaded at all (all require attempts failed and were collected in loadErrors). The message points at a known npm bug with optional dependencies (npm/cli#4828) where node_modules gets an incomplete set of platform-specific packages, so it advises a clean reinstall. All underlying per-platform require errors are chained via error.cause.

Solutions

1. Delete node_modules and package-lock.json, then run `npm i` again (per the npm/cli#4828 workaround)
2. Inspect error.cause chain to see which specific binding require failed and why
3. Install the platform-specific optional package explicitly, e.g. `npm i tokenizers-linux-x64-gnu` (or your platform's package)
4. If on an unsupported OS/arch, check `process.platform`/`process.arch`; the loader will never find a binding there (see 'Unsupported OS/architecture' load errors in the cause chain)

Example fix

// before: binding missing after partial install
const t = require('tokenizers') // throws Cannot find native binding...
// after
// rm -rf node_modules package-lock.json && npm i
const t = require('tokenizers') // works

function canLoadBinding() {
  try { require('tokenizers'); return true; } catch (err) {
    console.error('Binding load failed:', err.cause ?? err);
    return false;
  }
}

function isTokenizersLoaded(mod) {
  return !!mod && typeof mod === 'object' && typeof mod.encode !== 'undefined' || !!(mod && mod.AddedToken);
}

let tokenizers;
try {
  tokenizers = require('tokenizers');
} catch (err) {
  const firstCause = err.cause;
  console.error('Native binding missing. Underlying errors:', firstCause);
  console.error('Fix: rm -rf node_modules package-lock.json && npm i');
  throw err;
}

When it happens

Trigger: process.arch/process.platform matches no built binding, local .node files are missing, and platform packages like tokenizers-darwin-arm64/tokenizers-linux-x64-gnu etc. cannot be required, leaving nativeBinding null with loadErrors non-empty when bindings/node/index.js is required.

Common situations: npm bug #4828 dropping optional platform packages after `npm ci` from a lockfile created on another OS; running on an unusual platform (musl/Alpine, OpenHarmony) without the matching optional package; `npm i --no-optional`; switching Node/OS without reinstalling.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• Native binding package version mismatch, expected…
• Failed to load native binding
• WASI binding not found and NAPI_RS_FORCE_WASI is set to…

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/754d2282b8c33ff4. Report an issue: GitHub.

Appendix: source

Thrown at bindings/node/index.js:727

        loadErrors.push(err)
      }
    }
  }
  if (process.env.NAPI_RS_FORCE_WASI === 'error' && !wasiBinding) {
    const error = new Error('WASI binding not found and NAPI_RS_FORCE_WASI is set to error')
    error.cause = wasiBindingError
    throw error
  }
}

if (!nativeBinding) {
  if (loadErrors.length > 0) {
    throw new Error(
      `Cannot find native binding. ` +
        `npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). ` +
        'Please try `npm i` again after removing both package-lock.json and node_modules directory.',
      {
        cause: loadErrors.reduce((err, cur) => {
          cur.cause = err
          return cur
        }),
      },
    )
  }
  throw new Error(`Failed to load native binding`)
}

module.exports = nativeBinding
module.exports.AddedToken = nativeBinding.AddedToken
module.exports.BPE = nativeBinding.BPE
module.exports.Bpe = nativeBinding.Bpe
module.exports.Decoder = nativeBinding.Decoder
module.exports.Encoding = nativeBinding.Encoding
module.exports.JsEncoding = nativeBinding.JsEncoding
module.exports.Model = nativeBinding.Model
module.exports.Normalizer = nativeBinding.Normalizer

View on GitHub (pinned to 6cfd9d385c)