{"record":{"id":"4be52fbd2cad6b85","repo":"SonarSource/sonarqube","slug":"unable-to-lex-source-code-at-line-code-getl","errorCode":null,"errorMessage":"\"Unable to lex source code at line : \" + code.getLinePosition() + \" and column : \" + code.getColumnPosition()","messagePattern":"\"Unable to lex source code at line : \" \\+ code\\.getLinePosition\\(\\) \\+ \" and column : \" \\+ code\\.getColumnPosition\\(\\)","errorType":"exception","errorClass":"DuplicationsException","httpStatus":null,"severity":"error","filePath":"sonar-duplications/src/main/java/org/sonar/duplications/token/TokenChunker.java","lineNumber":52,"sourceCode":"    return new Builder();\n  }\n\n  private TokenChunker(Builder builder) {\n    this.channelDispatcher = builder.getChannelDispatcher();\n  }\n\n  public TokenQueue chunk(String sourceCode) {\n    return chunk(new StringReader(sourceCode));\n  }\n\n  public TokenQueue chunk(Reader reader) {\n    CodeReader code = new CodeReader(reader);\n    TokenQueue queue = new TokenQueue();\n    try {\n      channelDispatcher.consume(code, queue);\n      return queue;\n    } catch (Exception e) {\n      throw new DuplicationsException(\"Unable to lex source code at line : \" + code.getLinePosition() + \" and column : \" + code.getColumnPosition(), e);\n    }\n  }\n\n  /**\n   * Note that order is important, e.g.\n   * <code>token(\"A\").ignore(\"A\")</code> for the input string \"A\" will produce token, whereas\n   * <code>ignore(\"A\").token(\"A\")</code> will not.\n   */\n  public static final class Builder {\n\n    private ChannelDispatcher.Builder channelDispatcherBuilder = ChannelDispatcher.builder();\n\n    private Builder() {\n    }\n\n    public TokenChunker build() {\n      return new TokenChunker(this);\n    }","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/SonarSource/sonarqube/blob/184c821202192afc1c599fc912d0889b69fffa53/sonar-duplications/src/main/java/org/sonar/duplications/token/TokenChunker.java#L34-L70","documentation":"TokenChunker.chunk() lexes source code into a token queue for duplicate detection. If any exception (from the token channels or the CodeReader) escapes the channelDispatcher, it is rethrown as a DuplicationsException with the exact line and column position in the source where lexing failed. This indicates the input could not be tokenized with the configured language grammar.","triggerScenarios":"Calling chunk(Reader) with input that the configured token channels cannot parse — e.g. malformed characters, unexpected binary/encoding content, or a custom channel/bridge configuration that throws while consuming tokens.","commonSituations":"Feeding files with wrong encoding (binary data, UTF-16 read as UTF-8) into duplications analysis; custom language plugins with buggy token bridges; analyzing generated or corrupted source files.","solutions":["Look at the reported line/column in the exception to find the offending character or token in the source file.","Check the file encoding and re-encode the source to the expected encoding (e.g. UTF-8); rule out binary or corrupted files.","If a custom language plugin is in use, fix its token bridge/channel configuration (order matters) so the input can be tokenized.","Update the language plugin / SonarQube to a version handling your syntax."],"exampleFix":"// before: feeding reader directly, failing on bad encoding\nTokenQueue q = new TokenChunker(configuration).chunk(new FileReader(file));\n// after: read with explicit encoding so malformed bytes don't break lexing\nTokenQueue q = new TokenChunker(configuration)\n  .chunk(new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8));","handlingStrategy":"validation","validationCode":"// Ensure input is decodable text before lexing\nbyte[] bytes = Files.readAllBytes(file.toPath());\nCharsetDecoder dec = StandardCharsets.UTF_8.newDecoder()\n    .onMalformedInput(CodingErrorAction.REPORT);\ndec.decode(ByteBuffer.wrap(bytes)); // throws if not valid UTF-8","typeGuard":null,"tryCatchPattern":"try {\n  TokenQueue q = chunker.chunk(reader);\n} catch (DuplicationsException e) {\n  LOG.error(\"Lexing failed: \" + e.getMessage()); // line/column included\n}","preventionTips":["Always read source files with an explicit, correct charset","Exclude binary/generated files from analysis","Keep language plugin token bridges up to date","Log the reported line/column to quickly locate offending input"],"tags":["lexer","parsing","duplications","sonarqube"],"backgroundTag":"invalid-argument-format","analyzedSha":"184c821202192afc1c599fc912d0889b69fffa53","analyzedAt":"2026-09-09T12:23:51.573Z","contentChangedAt":"2026-09-09T12:23:51.573Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}