SonarSource/sonarqube · error · DuplicationsException

"Unable to lex source code at line : " +…

Error message

"Unable to lex source code at line : " + code.getLinePosition() + " and column : " + code.getColumnPosition()

What it means

TokenChunker.chunk() lexes source code into a token queue for duplicate detection. If any exception (from the token channels or the CodeReader) escapes the channelDispatcher, it is rethrown as a DuplicationsException with the exact line and column position in the source where lexing failed. This indicates the input could not be tokenized with the configured language grammar.

Solutions

1. Look at the reported line/column in the exception to find the offending character or token in the source file.
2. Check the file encoding and re-encode the source to the expected encoding (e.g. UTF-8); rule out binary or corrupted files.
3. If a custom language plugin is in use, fix its token bridge/channel configuration (order matters) so the input can be tokenized.
4. Update the language plugin / SonarQube to a version handling your syntax.

Example fix

// before: feeding reader directly, failing on bad encoding
TokenQueue q = new TokenChunker(configuration).chunk(new FileReader(file));
// after: read with explicit encoding so malformed bytes don't break lexing
TokenQueue q = new TokenChunker(configuration)
  .chunk(new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8));

// Ensure input is decodable text before lexing
byte[] bytes = Files.readAllBytes(file.toPath());
CharsetDecoder dec = StandardCharsets.UTF_8.newDecoder()
    .onMalformedInput(CodingErrorAction.REPORT);
dec.decode(ByteBuffer.wrap(bytes)); // throws if not valid UTF-8

try {
  TokenQueue q = chunker.chunk(reader);
} catch (DuplicationsException e) {
  LOG.error("Lexing failed: " + e.getMessage()); // line/column included
}

When it happens

Trigger: Calling chunk(Reader) with input that the configured token channels cannot parse — e.g. malformed characters, unexpected binary/encoding content, or a custom channel/bridge configuration that throws while consuming tokens.

Common situations: Feeding files with wrong encoding (binary data, UTF-16 read as UTF-8) into duplications analysis; custom language plugins with buggy token bridges; analyzing generated or corrupted source files.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Failed to parse duration threshold
• Failed to parse number of days
• A plugin is storing excessively large data in the following…
• A version event already exists on analysis
• Ad-hoc rules export failed after processing

AI-assisted analysis of SonarSource/sonarqube@184c821202 (2026-09-09). Data as JSON: /api/errors/4be52fbd2cad6b85. Report an issue: GitHub.

Appendix: source

Thrown at sonar-duplications/src/main/java/org/sonar/duplications/token/TokenChunker.java:52

    return new Builder();
  }

  private TokenChunker(Builder builder) {
    this.channelDispatcher = builder.getChannelDispatcher();
  }

  public TokenQueue chunk(String sourceCode) {
    return chunk(new StringReader(sourceCode));
  }

  public TokenQueue chunk(Reader reader) {
    CodeReader code = new CodeReader(reader);
    TokenQueue queue = new TokenQueue();
    try {
      channelDispatcher.consume(code, queue);
      return queue;
    } catch (Exception e) {
      throw new DuplicationsException("Unable to lex source code at line : " + code.getLinePosition() + " and column : " + code.getColumnPosition(), e);
    }
  }

  /**
   * Note that order is important, e.g.
   * <code>token("A").ignore("A")</code> for the input string "A" will produce token, whereas
   * <code>ignore("A").token("A")</code> will not.
   */
  public static final class Builder {

    private ChannelDispatcher.Builder channelDispatcherBuilder = ChannelDispatcher.builder();

    private Builder() {
    }

    public TokenChunker build() {
      return new TokenChunker(this);
    }

View on GitHub (pinned to 184c821202)