{"record":{"id":"14d791b02d4871ab","repo":"pentaho/pentaho-kettle","slug":"mysqlbulkloader-message-errorserializing","errorCode":"MySQLBulkLoader.Message.ERRORSERIALIZING","errorMessage":"MySQLBulkLoader.Message.ERRORSERIALIZING","messagePattern":"MySQLBulkLoader\\.Message\\.ERRORSERIALIZING","errorType":"exception","errorClass":"KettleException","httpStatus":null,"severity":"error","filePath":"plugins/mysql-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/mysqlbulkloader/MySQLBulkLoader.java","lineNumber":441,"sourceCode":"      // finally write a newline\n      //\n      data.fifoStream.write( data.newline );\n\n      if ( ( getLinesOutput() % 5000 ) == 0 ) {\n        data.fifoStream.flush();\n      }\n    } catch ( IOException e ) {\n      // If something went wrong with writing to the fifo, get the underlying error from MySQL\n      try {\n        logError( BaseMessages.getString( PKG, \"MySQLBulkLoader.Message.IOERROR\", this.threadWaitTimeText ) );\n        try {\n          data.sqlRunner.join( this.threadWaitTime );\n        } catch ( InterruptedException ex ) {\n          // Ignore errors\n        }\n        data.sqlRunner.checkExcn();\n      } catch ( Exception loadEx ) {\n        throw new KettleException( BaseMessages.getString( PKG, \"MySQLBulkLoader.Message.ERRORSERIALIZING\" ), loadEx );\n      }\n\n      // MySQL didn't finish, throw the generic \"Pipe\" exception.\n      throw new KettleException( BaseMessages.getString( PKG, \"MySQLBulkLoader.Message.ERRORSERIALIZING\" ), e );\n\n    } catch ( Exception e2 ) {\n      // Null pointer exceptions etc.\n      throw new KettleException( BaseMessages.getString( PKG, \"MySQLBulkLoader.Message.ERRORSERIALIZING\" ), e2 );\n    }\n  }\n\n  protected void verifyDatabaseConnection() throws KettleException {\n    // Confirming Database Connection is defined.\n    if ( meta.getDatabaseMeta() == null ) {\n      throw new KettleException( BaseMessages.getString( PKG, \"MySQLBulkLoaderMeta.GetSQL.NoConnectionDefined\" ) );\n    }\n  }\n","sourceCodeStart":423,"sourceCodeEnd":459,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/plugins/mysql-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/mysqlbulkloader/MySQLBulkLoader.java#L423-L459","documentation":"In MySQLBulkLoader.writeRowToBulk(), after writing a row to the named pipe and joining the SQL runner thread, any exception from data.sqlRunner (e.g. checkExcn() reporting the MySQL LOAD DATA failed) is rethrown as a KettleException with the localized message 'MySQLBulkLoader.Message.ERRORSERIALIZING' (\"Error serializing row to file\"). Despite the message text, this branch fires when the MySQL side of the bulk load failed, and the true cause is the wrapped loadEx.","triggerScenarios":"processRow -> writeRowToBulk: the row bytes are written to the pipe, data.sqlRunner.join() returns, and sqlRunner.checkExcn() (or any statement in the try) throws because the LOAD DATA INFILE on the MySQL server rejected the data or the runner thread died.","commonSituations":"MySQL rejects a value (bad encoding, wrong column count, too-long data for the column) during LOAD DATA; the bulk file/pipe path is misconfigured (e.g. Windows named pipe vs Unix FIFO mismatch); MySQL server killed the connection mid-load; charset mismatch between the transformation and the server.","solutions":["Inspect the wrapped cause (loadEx) — checkExcn() carries the real MySQL server error; fix the data or DDL it points to.","Verify the MySQL server can access the bulk load file/pipe path (secure_file_priv, local_infile, filesystem permissions).","Check column count/order and encodings in the MySQL Bulk Loader step settings match the target table.","Confirm the named pipe/FIFO setting matches the OS where the transformation runs (Unix FIFO vs Windows pipe).","Increase the stream/buffer settings or fall back to the regular MySQL table output step if bulk constraints can't be met."],"exampleFix":"// before: bulk loader fails because local_infile is disabled\nMySQLBulkLoader step -> LOAD DATA ... INFILE '/tmp/bulk.dat'\n// after: enable local infile on the server and connection\nSET GLOBAL local_infile = 1;\n// and on the connection URL:\njdbc:mysql://host/db?allowLoadLocalInfile=true","handlingStrategy":"validation","validationCode":"// Pre-flight before running the MySQL Bulk Loader transformation:\n// 1) server-side settings\nSELECT @@local_infile, @@secure_file_priv, @@wait_timeout;\n-- local_infile must be 1 (or use allowLoadLocalInfile=true on the JDBC URL)\n// 2) table shape matches step field mapping\nDESCRIBE target_table; -- compare column count/order/types with the step config\n// 3) charset\nSHOW VARIABLES LIKE 'character_set_server'; -- match transformation encoding","typeGuard":"boolean bulkLoaderReady(DatabaseMeta dbMeta, String tableName, String[] stepFields) {\n  try {\n    TableFields table = dbMeta.getTableFields(tableName); // column metadata\n    return table.size() == stepFields.length; // extend with type checks as needed\n  } catch (Exception e) { return false; }\n}","tryCatchPattern":"try {\n  transformation.start(); // runs the bulk loader\n} catch (KettleException e) {\n  Throwable root = e;\n  while (root.getCause() != null) root = root.getCause();\n  if (root.getMessage() != null && root.getMessage().contains(\"LOAD DATA\")) {\n    // MySQL rejected the data: fix row content / DDL, then retry\n  }\n  throw e;\n}","preventionTips":["Enable local_infile and allowLoadLocalInfile before bulk-load runs.","Keep field order, count, and types in the bulk loader step exactly aligned with the target table.","Match encodings (UTF-8) between transformation, bulk file, and MySQL server.","Verify secure_file_priv allows the directory used for the bulk file.","Always read the wrapped cause — ERRORSERIALIZING hides the real MySQL error."],"tags":["mysql","bulk-load","pdi","serialization","pipe"],"backgroundTag":"sql-query-failed","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}