{"record":{"id":"30d714497f60c10b","repo":"HandyOrg/HandyControl","slug":"tokenizerhelperemptytoken","errorCode":null,"errorMessage":"TokenizerHelperEmptyToken","messagePattern":"TokenizerHelperEmptyToken","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs","lineNumber":154,"sourceCode":"            ++newTokenLength;\n        }\n\n        // if quoteCount isn't zero we hit the end of the string\n        // before the ending quote\n        if (quoteCount > 0)\n        {\n            throw new InvalidOperationException(\"TokenizerHelperMissingEndQuote\");\n        }\n\n        ScanToNextToken(separator); // move so at the start of the nextToken for next call\n\n        // finally made it, update the _currentToken values\n        _currentTokenIndex = newTokenIndex;\n        _currentTokenLength = newTokenLength;\n\n        if (_currentTokenLength < 1)\n        {\n            throw new InvalidOperationException(\"TokenizerHelperEmptyToken\");\n        }\n\n        return true;\n    }\n\n    private void ScanToNextToken(char separator)\n    {\n        // if already at end of the string don't bother\n        if (_charIndex >= _strLen) return;\n\n        var currentChar = _str[_charIndex];\n\n        // check that the currentChar is a space or the separator.  If not\n        // we have an error. this can happen in the quote case\n        // that the char after the quotes string isn't a char.\n        if (currentChar != separator && !char.IsWhiteSpace(currentChar))\n        {\n            throw new InvalidOperationException(\"TokenizerHelperExtraDataEncountered\");","sourceCodeStart":136,"sourceCodeEnd":172,"githubUrl":"https://github.com/HandyOrg/HandyControl/blob/2c0875ebd67326e0c67282967e3e809c69282fee/src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs#L136-L172","documentation":"Sentinel validation error at the end of NextToken: tokenization completed but produced a zero-length token (_currentTokenLength < 1). This happens when the input has leading/consecutive separators or whitespace-only content where a token was expected — the helper treats an empty token as malformed input and throws instead of returning an empty string.","triggerScenarios":"Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:154 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Sanitize the input string before tokenizing: trim leading/trailing separators and whitespace, and collapse repeated separators so no empty tokens can be produced","Skip empty positions in the consuming loop: call NextToken only while LastSeparator/position indicates real content remains, or use a tokenizer API variant that skips empties","Wrap NextToken in try/catch for InvalidOperationException and abort parsing with a descriptive error naming the offending input"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2c0875ebd67326e0c67282967e3e809c69282fee","analyzedAt":"2026-09-14T14:45:29.754Z","contentChangedAt":"2026-09-14T14:45:29.754Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}