{"record":{"id":"1c097884b87ff904","repo":"lancedb/lancedb","slug":"invalid-language-code-lang","errorCode":null,"errorMessage":"Invalid language code {lang}","messagePattern":"Invalid language code (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/python/lancedb/table.py","lineNumber":3610,"sourceCode":"                \"language\": \"English\",\n                \"max_token_length\": None,\n                \"lower_case\": False,\n                \"stem\": False,\n                \"remove_stop_words\": False,\n                \"ascii_folding\": False,\n                \"ngram_min_length\": 3,\n                \"ngram_max_length\": 3,\n                \"prefix_only\": False,\n            }\n\n        # or it's with language stemming with pattern like \"en_stem\"\n        if len(tokenizer_name) != 7:\n            raise ValueError(f\"Invalid tokenizer name {tokenizer_name}\")\n        lang = tokenizer_name[:2]\n        if tokenizer_name[-5:] != \"_stem\":\n            raise ValueError(f\"Invalid tokenizer name {tokenizer_name}\")\n        if lang not in lang_mapping:\n            raise ValueError(f\"Invalid language code {lang}\")\n        return {\n            \"base_tokenizer\": \"simple\",\n            \"language\": lang_mapping[lang],\n            \"max_token_length\": 40,\n            \"lower_case\": True,\n            \"stem\": True,\n            \"remove_stop_words\": False,\n            \"ascii_folding\": False,\n            \"ngram_min_length\": 3,\n            \"ngram_max_length\": 3,\n            \"prefix_only\": False,\n        }\n\n    def add(\n        self,\n        data: DATA,\n        mode: AddMode = \"append\",\n        on_bad_vectors: OnBadVectorsType = \"error\",","sourceCodeStart":3592,"sourceCodeEnd":3628,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/python/python/lancedb/table.py#L3592-L3628","documentation":"Raised by the tokenizer-name validation helper in LanceTable.create when building a full-text-search index with a named stem tokenizer (e.g. 'en_stem'). The name must be exactly 7 characters of the form '<2-letter-lang>_stem', and the 2-letter language prefix must be one of the supported Lance tokenizer languages. The prefix was syntactically valid but not in the language mapping.","triggerScenarios":"Calling table.create_index with metric/'fts' and a tokenizer like 'xx_stem' where 'xx' is not a supported language code, or passing a 7-char string ending in _stem whose first two chars are not a known language (e.g. 'zz_stem').","commonSituations":"Typos in language codes ('eng_stem' instead of 'en_stem'), using ISO 639-2/639-3 codes where only 2-letter codes are accepted, or copy-pasted tokenizer names from other search libraries.","solutions":["Use a 2-letter language code from the supported set, e.g. 'en_stem' for English or 'de_stem' for German.","Check the lang_mapping in the tokenizer validation helper for the exact list of supported languages.","If your language is unsupported, omit the tokenizer (use the default simple tokenizer) or pre-stem the text yourself."],"exampleFix":"// before\nawait table.create_index(metric=\"fts\", config=FTS(with_position=False, tokenizer_name=\"eng_stem\"))\n// after\nawait table.create_index(metric=\"fts\", config=FTS(with_position=False, tokenizer_name=\"en_stem\"))","handlingStrategy":"validation","validationCode":"SUPPORTED_LANGS = {\"ar\",\"da\",\"de\",\"en\",\"es\",\"fi\",\"fr\",\"hu\",\"it\",\"nl\",\"no\",\"pt\",\"ro\",\"ru\",\"sv\",\"ta\",\"th\",\"tr\",\"zh\"}\nname = \"en_stem\"\nassert len(name) == 7 and name.endswith(\"_stem\") and name[:2] in SUPPORTED_LANGS","typeGuard":"def is_valid_stem_tokenizer(name: object) -> bool:\n    return isinstance(name, str) and len(name) == 7 and name.endswith(\"_stem\") and name[:2] in {\"ar\",\"da\",\"de\",\"en\",\"es\",\"fi\",\"fr\",\"hu\",\"it\",\"nl\",\"no\",\"pt\",\"ro\",\"ru\",\"sv\",\"ta\",\"th\",\"tr\",\"zh\"}","tryCatchPattern":null,"preventionTips":["Keep a constant list of supported 2-letter codes next to your index config.","Only use documented tokenizer_name values like 'en_stem', 'de_stem'.","Add a unit test asserting tokenizer names before creating FTS indexes."],"tags":["python","full-text-search","tokenizer","validation"],"backgroundTag":"invalid-enum-value","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}