{"record":{"id":"1351530956eea90d","repo":"sgl-project/sglang","slug":"pattern-must-contain-at-least-one-token","errorCode":null,"errorMessage":"pattern must contain at least one token","messagePattern":"pattern must contain at least one token","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/utils/token_sequence_matcher.py","lineNumber":21,"sourceCode":"# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom typing import Sequence\n\n\nclass TokenSequenceMatcher:\n    def __init__(self, pattern: Sequence[int]):\n        if not pattern:\n            raise ValueError(\"pattern must contain at least one token\")\n        self.pattern = tuple(pattern)\n        self.prefix_lengths = self._build_prefix_lengths()\n\n    def _build_prefix_lengths(self) -> tuple[int, ...]:\n        prefix_lengths = [0] * len(self.pattern)\n        matched = 0\n        for index in range(1, len(self.pattern)):\n            while matched > 0 and self.pattern[index] != self.pattern[matched]:\n                matched = prefix_lengths[matched - 1]\n            if self.pattern[index] == self.pattern[matched]:\n                matched += 1\n            prefix_lengths[index] = matched\n        return tuple(prefix_lengths)\n\n    def __len__(self) -> int:\n        return len(self.pattern)\n\n    def advance(self, matched: int, token: int) -> int:","sourceCodeStart":3,"sourceCodeEnd":39,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/utils/token_sequence_matcher.py#L3-L39","documentation":"TokenSequenceMatcher implements KMP-style matching over token id sequences and requires a non-empty pattern to build its prefix-length table. An empty pattern has no valid failure function, so __init__ rejects it immediately.","triggerScenarios":"Constructing TokenSequenceMatcher([]) or TokenSequenceMatcher(()) — e.g. from an empty stop-token list or a filtered list that removed every token.","commonSituations":"Dynamic patterns built from user input or config that can legitimately be empty; upstream filter/list comprehension yielding zero elements.","solutions":["Guard before construction: if pattern: matcher = TokenSequenceMatcher(pattern)","Skip matching entirely when the pattern list is empty","Validate pattern lists at config-load time and require >=1 token"],"exampleFix":"# before\nmatcher = TokenSequenceMatcher(stop_tokens)  # may be []\n# after\nmatcher = TokenSequenceMatcher(stop_tokens) if stop_tokens else None\nif matcher and matcher.search(stream): ...","handlingStrategy":"validation","validationCode":"if not pattern:\n    return None  # or skip matching\nmatcher = TokenSequenceMatcher(pattern)","typeGuard":"def has_tokens(p) -> bool:\n    return len(p) > 0","tryCatchPattern":null,"preventionTips":["Treat an empty stop/pattern list as 'no matching', not as a matcher","Validate pattern lists at config-load time"],"tags":["sglang","tokens","pattern-matching","validation","kmp"],"backgroundTag":"empty-pattern-validation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}