pentaho/pentaho-kettle · error · KettleException

MySQLBulkLoader.Message.ERRORSERIALIZING

MySQLBulkLoader.Message.ERRORSERIALIZING

Error message

MySQLBulkLoader.Message.ERRORSERIALIZING

What it means

In MySQLBulkLoader.writeRowToBulk(), after writing a row to the named pipe and joining the SQL runner thread, any exception from data.sqlRunner (e.g. checkExcn() reporting the MySQL LOAD DATA failed) is rethrown as a KettleException with the localized message 'MySQLBulkLoader.Message.ERRORSERIALIZING' ("Error serializing row to file"). Despite the message text, this branch fires when the MySQL side of the bulk load failed, and the true cause is the wrapped loadEx.

Solutions

  1. Inspect the wrapped cause (loadEx) — checkExcn() carries the real MySQL server error; fix the data or DDL it points to.
  2. Verify the MySQL server can access the bulk load file/pipe path (secure_file_priv, local_infile, filesystem permissions).
  3. Check column count/order and encodings in the MySQL Bulk Loader step settings match the target table.
  4. Confirm the named pipe/FIFO setting matches the OS where the transformation runs (Unix FIFO vs Windows pipe).
  5. Increase the stream/buffer settings or fall back to the regular MySQL table output step if bulk constraints can't be met.

Example fix

// before: bulk loader fails because local_infile is disabled
MySQLBulkLoader step -> LOAD DATA ... INFILE '/tmp/bulk.dat'
// after: enable local infile on the server and connection
SET GLOBAL local_infile = 1;
// and on the connection URL:
jdbc:mysql://host/db?allowLoadLocalInfile=true
Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight before running the MySQL Bulk Loader transformation:
// 1) server-side settings
SELECT @@local_infile, @@secure_file_priv, @@wait_timeout;
-- local_infile must be 1 (or use allowLoadLocalInfile=true on the JDBC URL)
// 2) table shape matches step field mapping
DESCRIBE target_table; -- compare column count/order/types with the step config
// 3) charset
SHOW VARIABLES LIKE 'character_set_server'; -- match transformation encoding

Type guard

boolean bulkLoaderReady(DatabaseMeta dbMeta, String tableName, String[] stepFields) {
  try {
    TableFields table = dbMeta.getTableFields(tableName); // column metadata
    return table.size() == stepFields.length; // extend with type checks as needed
  } catch (Exception e) { return false; }
}

Try / catch

try {
  transformation.start(); // runs the bulk loader
} catch (KettleException e) {
  Throwable root = e;
  while (root.getCause() != null) root = root.getCause();
  if (root.getMessage() != null && root.getMessage().contains("LOAD DATA")) {
    // MySQL rejected the data: fix row content / DDL, then retry
  }
  throw e;
}

Prevention

When it happens

Trigger: processRow -> writeRowToBulk: the row bytes are written to the pipe, data.sqlRunner.join() returns, and sqlRunner.checkExcn() (or any statement in the try) throws because the LOAD DATA INFILE on the MySQL server rejected the data or the runner thread died.

Common situations: MySQL rejects a value (bad encoding, wrong column count, too-long data for the column) during LOAD DATA; the bulk file/pipe path is misconfigured (e.g. Windows named pipe vs Unix FIFO mismatch); MySQL server killed the connection mid-load; charset mismatch between the transformation and the server.

Understand the failure class

Background: "query failed", "%w: SQL error" — wrapped database query errors in Go libraries explained — this error's family across 3 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/14d791b02d4871ab. Report an issue: GitHub.

Appendix: source

Thrown at plugins/mysql-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/mysqlbulkloader/MySQLBulkLoader.java:441

      // finally write a newline
      //
      data.fifoStream.write( data.newline );

      if ( ( getLinesOutput() % 5000 ) == 0 ) {
        data.fifoStream.flush();
      }
    } catch ( IOException e ) {
      // If something went wrong with writing to the fifo, get the underlying error from MySQL
      try {
        logError( BaseMessages.getString( PKG, "MySQLBulkLoader.Message.IOERROR", this.threadWaitTimeText ) );
        try {
          data.sqlRunner.join( this.threadWaitTime );
        } catch ( InterruptedException ex ) {
          // Ignore errors
        }
        data.sqlRunner.checkExcn();
      } catch ( Exception loadEx ) {
        throw new KettleException( BaseMessages.getString( PKG, "MySQLBulkLoader.Message.ERRORSERIALIZING" ), loadEx );
      }

      // MySQL didn't finish, throw the generic "Pipe" exception.
      throw new KettleException( BaseMessages.getString( PKG, "MySQLBulkLoader.Message.ERRORSERIALIZING" ), e );

    } catch ( Exception e2 ) {
      // Null pointer exceptions etc.
      throw new KettleException( BaseMessages.getString( PKG, "MySQLBulkLoader.Message.ERRORSERIALIZING" ), e2 );
    }
  }

  protected void verifyDatabaseConnection() throws KettleException {
    // Confirming Database Connection is defined.
    if ( meta.getDatabaseMeta() == null ) {
      throw new KettleException( BaseMessages.getString( PKG, "MySQLBulkLoaderMeta.GetSQL.NoConnectionDefined" ) );
    }
  }

View on GitHub (pinned to f3058517a1)