{"record":{"id":"448ab97cc26ad05a","repo":"dotnet/machinelearning","slug":"failed-to-insert-key-zero-length-key","errorCode":null,"errorMessage":"failed to insert key: zero-length key","messagePattern":"failed to insert key: zero-length key","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs","lineNumber":346,"sourceCode":"\n        public void Finish()\n        {\n            Flush(0);\n\n            _units[0] = _nodes[0].Unit;\n            _labels[0] = _nodes[0].Label;\n            _isIntersections.Build();\n        }\n\n        public void Insert(ReadOnlySpan<byte> key, int length, int value)\n        {\n            if (value < 0)\n            {\n                throw new ArgumentException(\"failed to insert key: negative value\");\n            }\n            else if (length == 0)\n            {\n                throw new ArgumentException(\"failed to insert key: zero-length key\");\n            }\n\n            uint id = 0;\n            int keyPos = 0;\n\n            for (; keyPos <= length; ++keyPos)\n            {\n                uint childId = _nodes[(int)id].Child;\n                if (childId == 0)\n                {\n                    break;\n                }\n\n                byte keyLabel = key[keyPos];\n                if (keyPos < length && keyLabel == 0)\n                {\n                    throw new InvalidOperationException(\"failed to insert key: invalid null character\");\n                }","sourceCodeStart":328,"sourceCodeEnd":364,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs#L328-L364","documentation":"DoubleArrayTrie.Insert throws this ArgumentException when the key length passed is zero. An empty key cannot be represented in the double-array structure, so the library refuses it up front. Like the negative-value check, this fires while building the trie from a vocabulary.","triggerScenarios":"Calling Insert with length == 0, e.g. BuildDawg encountering an empty string entry in the vocabulary (empty token in the sentencepiece vocab or a splitting bug producing empty pieces).","commonSituations":"A malformed vocab file with an empty token line, a text-splitting routine that emits empty substrings, or a loop bug passing zero length.","solutions":["Filter out empty keys before inserting: skip entries where the piece is the empty string.","Fix the splitting/parsing code that produces empty tokens.","Check the vocab file for blank lines or empty pieces and clean it.","Add a guard in the builder loop: if (piece.Length == 0) continue;"],"exampleFix":"// before\nforeach (var piece in vocab.Keys)\n    trie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id++);\n// after\nforeach (var piece in vocab.Keys.Where(p => p.Length > 0))\n    trie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id++);","handlingStrategy":"validation","validationCode":"if (piece.Length == 0) return; // or throw\ntrie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id);","typeGuard":"static bool IsInsertableKey(ReadOnlySpan<byte> key) => key.Length > 0 && !key.Contains((byte)0);","tryCatchPattern":"try { trie.Insert(key, key.Length, id); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"zero-length\"))\n{ /* skip or log the empty key */ }","preventionTips":["Filter empty tokens before building the trie","Check vocab files for blank lines/empty pieces","Assert piece.Length > 0 in the builder loop"],"tags":["trie","empty-key","validation","tokenizer"],"backgroundTag":"empty-required-field","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}