huggingface/tokenizers · error · Error

Failed to load native binding

Error message

Failed to load native binding

What it means

Thrown when requireNative() found no binding (nativeBinding is falsy) but loadErrors is empty — i.e., the loader cannot even record why loading failed. Unlike the 'Cannot find native binding' error, it carries no npm-optional-dependency hint and no cause, making it the generic 'something silently prevented binding selection' failure at module load.

Solutions

  1. Re-run in plain Node (no require interception/mocks) to see if the error persists and inspect earlier console output
  2. Delete node_modules and package-lock.json and reinstall, as with the 'Cannot find native binding' case
  3. Verify you're on a supported platform/arch (check process.platform and process.arch); unsupported combos push descriptive errors into loadErrors — if you see this instead, something interfered with loading
  4. Check bundler/test configs (webpack resolve, jest moduleNameMapper) for overrides of the binding modules

Example fix

// before (jest mock returning undefined)
jest.mock('tokenizers-linux-x64-gnu', () => undefined)
// after
jest.mock('tokenizers-linux-x64-gnu', () => require('./__mocks__/binding.js'))
Defensive patterns

Strategy: try-catch

Validate before calling

let tokenizers;
try { tokenizers = require('tokenizers'); } catch {}
if (!tokenizers) {
  console.error('Failed to load native binding; check platform support and bundler overrides');
}

Type guard

function hasNativeBinding(mod) {
  return !!mod && typeof mod === 'object' && Object.keys(mod).length > 0;
}

Try / catch

try {
  const tokenizers = require('tokenizers');
} catch (err) {
  if (err.message === 'Failed to load native binding') {
    console.error('No load errors recorded — check for require mocks/overrides or unsupported platform:', process.platform, process.arch);
  }
  throw err;
}

Prevention

When it happens

Trigger: Module-level code in bindings/node/index.js:734 executes with nativeBinding falsy and loadErrors.length === 0 — e.g. an environment where no platform branch set loadErrors yet every require returned falsy, or a monkey-patched/intercepted require returning undefined instead of throwing.

Common situations: Custom bundler/mocking setups (e.g. jest moduleNameMapper) that replace require with a function returning undefined; exotic embedded environments where no platform branch matched; unusual loader interception tools that swallow require errors.

Related errors

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/aca802a24db2368a. Report an issue: GitHub.

Appendix: source

Thrown at bindings/node/index.js:734

    throw error
  }
}

if (!nativeBinding) {
  if (loadErrors.length > 0) {
    throw new Error(
      `Cannot find native binding. ` +
        `npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). ` +
        'Please try `npm i` again after removing both package-lock.json and node_modules directory.',
      {
        cause: loadErrors.reduce((err, cur) => {
          cur.cause = err
          return cur
        }),
      },
    )
  }
  throw new Error(`Failed to load native binding`)
}

module.exports = nativeBinding
module.exports.AddedToken = nativeBinding.AddedToken
module.exports.BPE = nativeBinding.BPE
module.exports.Bpe = nativeBinding.Bpe
module.exports.Decoder = nativeBinding.Decoder
module.exports.Encoding = nativeBinding.Encoding
module.exports.JsEncoding = nativeBinding.JsEncoding
module.exports.Model = nativeBinding.Model
module.exports.Normalizer = nativeBinding.Normalizer
module.exports.PreTokenizer = nativeBinding.PreTokenizer
module.exports.Processor = nativeBinding.Processor
module.exports.Tokenizer = nativeBinding.Tokenizer
module.exports.Trainer = nativeBinding.Trainer
module.exports.Unigram = nativeBinding.Unigram
module.exports.WordLevel = nativeBinding.WordLevel
module.exports.WordPiece = nativeBinding.WordPiece

View on GitHub (pinned to 6cfd9d385c)