pentaho/pentaho-kettle · error · KettleException

S3CsvInput.Exception.ErrorPreparingParallelRun

S3CsvInput.Exception.ErrorPreparingParallelRun

Error message

S3CsvInput.Exception.ErrorPreparingParallelRun

What it means

S3CsvInput.prepareToRunInParallel computes file sizes and block offsets so multiple step copies can split a large S3 CSV file. Any exception during that setup (S3 metadata fetch failures, null sizes, bad filenames) is rethrown as a KettleException with message key S3CsvInput.Exception.ErrorPreparingParallelRun.

Solutions

1. Verify each file in the step's file list exists in the bucket and credentials can read it (aws s3 ls / head-object)
2. If filenames come from a previous step, confirm rows actually arrive before this step executes
3. Check the wrapped cause (e.getCause()) in the KettleException log for the real failure (NoSuchBucket, 403, etc.)
4. Disable or correctly configure parallel execution if the step copy receives no files

Example fix

// before
// filenames resolved from previous step may be empty
prepareToRunInParallel( layouts );
// after
if ( data.filenames == null || data.filenames.length == 0 ) {
  logBasic( "No files to process; skipping parallel preparation" );
  return;
}
prepareToRunInParallel( layouts );

for ( String f : filenames ) {
  if ( f == null || f.isEmpty() ) throw new IllegalArgumentException( "empty filename" );
  if ( !s3.doesObjectExist( bucket, key( f ) ) ) throw new IllegalStateException( "missing: " + f );
}

try {
  prepareToRunInParallel( layouts );
} catch ( KettleException e ) {
  logError( "parallel prep failed", e.getCause() );
  throw e;
}

When it happens

Trigger: Running the S3 CSV Input step in 'parallel run' mode when data.filenames lookup, S3 object metadata retrieval (size), or the size/block calculation throws — e.g. object missing, credentials invalid, or filenames list empty/null.

Common situations: Misconfigured parallel/clustered transformation with a non-existent S3 key; expired S3 credentials; running the step with 0 input files; region mismatch making head-object calls fail.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• S3CsvInput.Exception.FilenameFieldNotFound
• Exception reading line using NIO
• A connection of type PALO is expected
• A connection of type PALO is expected
• A deadlock was detected between steps

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/477d41b407a9111d. Report an issue: GitHub.

Appendix: source

Thrown at plugins/s3csvinput/core/src/main/java/org/pentaho/di/trans/steps/s3csvinput/S3CsvInput.java:187

          data.startFilenr = i;


          // How many bytes do we skip in that first file?
          //
          if ( data.startPosition == 0 ) {
            data.bytesToSkipInFirstFile = 0L;
          } else {
            data.bytesToSkipInFirstFile = data.startPosition - totalFileSize;
          }

          break;
        }
        totalFileSize += size;
      }

      logBasic( Messages.getString( "S3CsvInput.Log.ParallelFileNrAndPositionFeedback", data.filenames[data.filenr], Long.toString( data.fileSizes.get( data.filenr ) ), Long.toString( data.bytesToSkipInFirstFile ), Long.toString( data.blockToRead ) ) );
    } catch ( Exception e ) {
      throw new KettleException( Messages.getString( "S3CsvInput.Exception.ErrorPreparingParallelRun" ), e );
    }
  }

  private void getFilenamesFromPreviousSteps() throws KettleException {
    List<String> filenames = new ArrayList<String>();
    boolean firstRow = true;
    int index = -1;
    Object[] row = getRow();
    while ( row != null ) {

      if ( firstRow ) {
        firstRow = false;

        // Get the filename field index...
        //
        String filenameField = environmentSubstitute( meta.getFilenameField() );
        index = getInputRowMeta().indexOfValue( filenameField );
        if ( index < 0 ) {

View on GitHub (pinned to f3058517a1)