apache/beam · error · RuntimeException

Some tokens are missing from the splits. This should not…

Error message

Some tokens are missing from the splits. This should not happen.

What it means

After generating splits, generateSplits() asserts that the spans of all produced RingRanges sum exactly to the partitioner's total range size. If not, an internal invariant was violated (tokens lost or double-counted in the split algorithm) and a RuntimeException is thrown.

Solutions

1. Report/reproduce with the exact ring token list (nodetool describering) and partitioner.
2. Verify you are using an unmodified SplitGenerator from your Beam version; upgrade to a newer Beam release.
3. As a workaround, reduce desired split count or read without explicit split configuration if the issue is split-count dependent.

// pin a Beam version known-good for CassandraIO and avoid custom forked SplitGenerator
System.out.println(org.apache.beam.sdk.Pipeline.class.getPackage().getImplementationVersion());

try { ... } catch (RuntimeException e) { if (e.getMessage().contains("Some tokens are missing")) { /* report bug with ring dump */ } throw e; }

When it happens

Trigger: A bug or inconsistent numeric edge case in split arithmetic — total of split spans != rangeSize — during CassandraIO read split planning.

Common situations: Modified/custom SplitGenerator code; integer/precision edge cases around the wrap-around range; unexpected ring data interacting with the algorithm.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• 2xx codes should not be exceptions. Got status code
• Column has array of arrays which is prohibited in Spanner.
• with no or
• Tokens ( , ) not in range of
• Tokens ( , ): two nodes have the same token

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f4ee42c9db334266. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/SplitGenerator.java:141

        // See https://issues.apache.org/jira/browse/CASSANDRA-14684
        endpointTokens.add(
            token.equals(BigInteger.valueOf(Long.MIN_VALUE)) ? token.add(BigInteger.ONE) : token);
      }

      // Append the splits between the endpoints
      for (int j = 0; j < splitCount; j++) {
        splits.add(RingRange.of(endpointTokens.get(j), endpointTokens.get(j + 1)));
        LOG.debug("Split #{}: [{},{})", j + 1, endpointTokens.get(j), endpointTokens.get(j + 1));
      }
    }

    BigInteger total = BigInteger.ZERO;
    for (RingRange split : splits) {
      BigInteger size = split.span(rangeSize);
      total = total.add(size);
    }
    if (!total.equals(rangeSize)) {
      throw new RuntimeException(
          "Some tokens are missing from the splits. " + "This should not happen.");
    }
    return coalesceSplits(getTargetSplitSize(totalSplitCount), splits);
  }

  private boolean isInRange(BigInteger token) {
    return !(token.compareTo(rangeMin) < 0 || token.compareTo(rangeMax) > 0);
  }

  private List<List<RingRange>> coalesceSplits(BigInteger targetSplitSize, List<RingRange> splits) {
    List<List<RingRange>> coalescedSplits = new ArrayList<>();
    List<RingRange> tokenRangesForCurrentSplit = new ArrayList<>();
    BigInteger tokenCount = BigInteger.ZERO;

    for (RingRange tokenRange : splits) {
      if (tokenRange.span(rangeSize).add(tokenCount).compareTo(targetSplitSize) > 0
          && !tokenRangesForCurrentSplit.isEmpty()) {
        // enough tokens in that segment

View on GitHub (pinned to 12126d8942)