pentaho/pentaho-kettle · error · KettleStepException

Internal error: the referenced partition nr '" +…

Error message

Internal error: the referenced partition nr '" + partitionNr + "' is higher than the maximum of '" + ( partitionNrRowSetList.length - 1 ) + ".  The available row sets are: {" + rowsets + "}

What it means

After computing the partition number, BaseStep indexes into partitionNrRowSetList to find the target rowset. If partitionNr exceeds the array bounds, this internal-error KettleStepException is thrown, listing all available rowsets. It means the partition number produced by the partitioner is larger than the number of target step copies/rowsets allocated.

Solutions

  1. Ensure the number of step copies of the target (partitioned) step equals the partitioner's configured number of partitions/slaves.
  2. Re-save the transformation so partition metadata and step copies are consistent.
  3. Verify identical transformation XML on master and all slaves in clustered runs.
  4. Catch KettleStepException and log partitionNr plus available rowsets (included in the message) to spot the mismatch.

Example fix

// before: step copies = 2, partitioner ' nrPartitions' = 4
// after: set both consistently
partitioner.setNrPartitions(2); // and give the target step 2 copies
Defensive patterns

Strategy: validation

Validate before calling

// Ensure target step copies equal partitioner count before execution
int copies = transMeta.findStep(nextStepName).getCopies();
int parts = partitionerMeta.getPartitioner().getNrPartitions();
if (copies != parts) throw new IllegalStateException("copies=" + copies + " != nrPartitions=" + parts);

Try / catch

try {
  trans.execute(null);
} catch (KettleStepException e) {
  if (e.getMessage().contains("is higher than the maximum")) {
    logError("Partition count vs step copies mismatch: " + e.getMessage(), e);
  } else { throw e; }
}

Prevention

When it happens

Trigger: partitionNrRowSetList[partitionNr] is accessed where partitionNr >= partitionNrRowSetList.length — e.g. the partitioner was configured for N partitions but the next step only has fewer copies running, or the partition metadata changed at runtime while rowsets were built for a smaller count.

Common situations: Changing the number of step copies (nrPartitions) after previously saving the transformation; running a transformation designed for more partitions in a cluster/slave environment with fewer allocated; stale transformation metadata between cluster master and slaves.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/0c0822290ec9786c. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/step/BaseStep.java:1449

            }
            partitionNrRowSetList[ partNr ] = outputRowSet;
          } catch ( NullPointerException e ) {
            throw ( e );
          }
        }
      }

      // OK, now get the target partition based on the partition nr...
      // This should be very fast
      //
      if ( partitionNr < partitionNrRowSetList.length ) {
        selectedRowSet = partitionNrRowSetList[ partitionNr ];
      } else {
        String rowsets = "";
        for ( RowSet rowSet : partitionNrRowSetList ) {
          rowsets += "[" + rowSet.toString() + "] ";
        }
        throw new KettleStepException( "Internal error: the referenced partition nr '"
          + partitionNr + "' is higher than the maximum of '" + ( partitionNrRowSetList.length - 1 )
          + ".  The available row sets are: {" + rowsets + "}" );
      }

      if ( selectedRowSet == null ) {
        logBasic( BaseMessages.getString( PKG, "BaseStep.TargetRowsetIsNotAvailable", partitionNr ) );
      } else {
        // Wait
        putRowToRowSet( selectedRowSet, rowMeta, row );
        incrementLinesWritten();

        if ( log.isRowLevel() ) {
          try {
            logRowlevel(
              "Partitioned #" + partitionNr + " to " + selectedRowSet + ", row=" + rowMeta.getString( row ) );
          } catch ( KettleValueException e ) {
            throw new KettleStepException( e );
          }

View on GitHub (pinned to f3058517a1)