lancedb/lancedb · error · ValueError

Invalid language code

Error message

Invalid language code {lang}

What it means

Raised by the tokenizer-name validation helper in LanceTable.create when building a full-text-search index with a named stem tokenizer (e.g. 'en_stem'). The name must be exactly 7 characters of the form '<2-letter-lang>_stem', and the 2-letter language prefix must be one of the supported Lance tokenizer languages. The prefix was syntactically valid but not in the language mapping.

Solutions

1. Use a 2-letter language code from the supported set, e.g. 'en_stem' for English or 'de_stem' for German.
2. Check the lang_mapping in the tokenizer validation helper for the exact list of supported languages.
3. If your language is unsupported, omit the tokenizer (use the default simple tokenizer) or pre-stem the text yourself.

Example fix

// before
await table.create_index(metric="fts", config=FTS(with_position=False, tokenizer_name="eng_stem"))
// after
await table.create_index(metric="fts", config=FTS(with_position=False, tokenizer_name="en_stem"))

SUPPORTED_LANGS = {"ar","da","de","en","es","fi","fr","hu","it","nl","no","pt","ro","ru","sv","ta","th","tr","zh"}
name = "en_stem"
assert len(name) == 7 and name.endswith("_stem") and name[:2] in SUPPORTED_LANGS

def is_valid_stem_tokenizer(name: object) -> bool:
    return isinstance(name, str) and len(name) == 7 and name.endswith("_stem") and name[:2] in {"ar","da","de","en","es","fi","fr","hu","it","nl","no","pt","ro","ru","sv","ta","th","tr","zh"}

When it happens

Trigger: Calling table.create_index with metric/'fts' and a tokenizer like 'xx_stem' where 'xx' is not a supported language code, or passing a 7-char string ending in _stem whose first two chars are not a known language (e.g. 'zz_stem').

Common situations: Typos in language codes ('eng_stem' instead of 'en_stem'), using ISO 639-2/639-3 codes where only 2-letter codes are accepted, or copy-pasted tokenizer names from other search libraries.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• a Function environment is pip or conda, not both
• All elements in vector_results should be of the same type
• All elements in vector_results should be of the same type
• All elements in vector_results should be of the same type
• blocks_per_epoch='auto' cannot estimate an empty dataset

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/1c097884b87ff904. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/table.py:3610

                "language": "English",
                "max_token_length": None,
                "lower_case": False,
                "stem": False,
                "remove_stop_words": False,
                "ascii_folding": False,
                "ngram_min_length": 3,
                "ngram_max_length": 3,
                "prefix_only": False,
            }

        # or it's with language stemming with pattern like "en_stem"
        if len(tokenizer_name) != 7:
            raise ValueError(f"Invalid tokenizer name {tokenizer_name}")
        lang = tokenizer_name[:2]
        if tokenizer_name[-5:] != "_stem":
            raise ValueError(f"Invalid tokenizer name {tokenizer_name}")
        if lang not in lang_mapping:
            raise ValueError(f"Invalid language code {lang}")
        return {
            "base_tokenizer": "simple",
            "language": lang_mapping[lang],
            "max_token_length": 40,
            "lower_case": True,
            "stem": True,
            "remove_stop_words": False,
            "ascii_folding": False,
            "ngram_min_length": 3,
            "ngram_max_length": 3,
            "prefix_only": False,
        }

    def add(
        self,
        data: DATA,
        mode: AddMode = "append",
        on_bad_vectors: OnBadVectorsType = "error",

View on GitHub (pinned to c7b051aff7)