{"record":{"id":"0c0822290ec9786c","repo":"pentaho/pentaho-kettle","slug":"internal-error-the-referenced-partition-nr-partitionnr-is","errorCode":null,"errorMessage":"Internal error: the referenced partition nr '\" + partitionNr + \"' is higher than the maximum of '\" + ( partitionNrRowSetList.length - 1 ) + \".  The available row sets are: {\" + rowsets + \"}","messagePattern":"Internal error: the referenced partition nr '\" \\+ partitionNr \\+ \"' is higher than the maximum of '\" \\+ \\( partitionNrRowSetList\\.length - 1 \\) \\+ \"\\.  The available row sets are: (.+?)","errorType":"exception","errorClass":"KettleStepException","httpStatus":null,"severity":"error","filePath":"engine/src/main/java/org/pentaho/di/trans/step/BaseStep.java","lineNumber":1449,"sourceCode":"            }\n            partitionNrRowSetList[ partNr ] = outputRowSet;\n          } catch ( NullPointerException e ) {\n            throw ( e );\n          }\n        }\n      }\n\n      // OK, now get the target partition based on the partition nr...\n      // This should be very fast\n      //\n      if ( partitionNr < partitionNrRowSetList.length ) {\n        selectedRowSet = partitionNrRowSetList[ partitionNr ];\n      } else {\n        String rowsets = \"\";\n        for ( RowSet rowSet : partitionNrRowSetList ) {\n          rowsets += \"[\" + rowSet.toString() + \"] \";\n        }\n        throw new KettleStepException( \"Internal error: the referenced partition nr '\"\n          + partitionNr + \"' is higher than the maximum of '\" + ( partitionNrRowSetList.length - 1 )\n          + \".  The available row sets are: {\" + rowsets + \"}\" );\n      }\n\n      if ( selectedRowSet == null ) {\n        logBasic( BaseMessages.getString( PKG, \"BaseStep.TargetRowsetIsNotAvailable\", partitionNr ) );\n      } else {\n        // Wait\n        putRowToRowSet( selectedRowSet, rowMeta, row );\n        incrementLinesWritten();\n\n        if ( log.isRowLevel() ) {\n          try {\n            logRowlevel(\n              \"Partitioned #\" + partitionNr + \" to \" + selectedRowSet + \", row=\" + rowMeta.getString( row ) );\n          } catch ( KettleValueException e ) {\n            throw new KettleStepException( e );\n          }","sourceCodeStart":1431,"sourceCodeEnd":1467,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/engine/src/main/java/org/pentaho/di/trans/step/BaseStep.java#L1431-L1467","documentation":"After computing the partition number, BaseStep indexes into partitionNrRowSetList to find the target rowset. If partitionNr exceeds the array bounds, this internal-error KettleStepException is thrown, listing all available rowsets. It means the partition number produced by the partitioner is larger than the number of target step copies/rowsets allocated.","triggerScenarios":"partitionNrRowSetList[partitionNr] is accessed where partitionNr >= partitionNrRowSetList.length — e.g. the partitioner was configured for N partitions but the next step only has fewer copies running, or the partition metadata changed at runtime while rowsets were built for a smaller count.","commonSituations":"Changing the number of step copies (nrPartitions) after previously saving the transformation; running a transformation designed for more partitions in a cluster/slave environment with fewer allocated; stale transformation metadata between cluster master and slaves.","solutions":["Ensure the number of step copies of the target (partitioned) step equals the partitioner's configured number of partitions/slaves.","Re-save the transformation so partition metadata and step copies are consistent.","Verify identical transformation XML on master and all slaves in clustered runs.","Catch KettleStepException and log partitionNr plus available rowsets (included in the message) to spot the mismatch."],"exampleFix":"// before: step copies = 2, partitioner ' nrPartitions' = 4\n// after: set both consistently\npartitioner.setNrPartitions(2); // and give the target step 2 copies","handlingStrategy":"validation","validationCode":"// Ensure target step copies equal partitioner count before execution\nint copies = transMeta.findStep(nextStepName).getCopies();\nint parts = partitionerMeta.getPartitioner().getNrPartitions();\nif (copies != parts) throw new IllegalStateException(\"copies=\" + copies + \" != nrPartitions=\" + parts);","typeGuard":null,"tryCatchPattern":"try {\n  trans.execute(null);\n} catch (KettleStepException e) {\n  if (e.getMessage().contains(\"is higher than the maximum\")) {\n    logError(\"Partition count vs step copies mismatch: \" + e.getMessage(), e);\n  } else { throw e; }\n}","preventionTips":["Keep nrPartitions and target step copies synchronized","Re-save transformations after changing partition counts","Verify identical transformation definitions on all cluster nodes"],"tags":["partitioning","index-out-of-range","cluster"],"backgroundTag":"index-out-of-bounds","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}