{"record":{"id":"e9c5cce072b29fc9","repo":"pentaho/pentaho-kettle","slug":"error-serializing-rows-of-data-to-the-copy-command","errorCode":null,"errorMessage":"Error serializing rows of data to the COPY command","messagePattern":"Error serializing rows of data to the COPY command","errorType":"exception","errorClass":"KettleException","httpStatus":null,"severity":"error","filePath":"plugins/postgresql-db-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/pgbulkloader/PGBulkLoader.java","lineNumber":423,"sourceCode":"                pgCopyOut.write( (byte[]) valueData );\n              } else {\n                BigDecimal big = valueMeta.getBigNumber( valueData );\n                if ( big != null ) {\n                  pgCopyOut.write( big.toString().getBytes( clientEncoding ) );\n                }\n              }\n              break;\n            default:\n              throw new KettleException( \"PGBulkLoader doesn't handle the type \" + valueMeta.getTypeDesc() );\n          }\n        }\n      }\n\n      // Now write a newline\n      //\n      pgCopyOut.write( data.newline );\n    } catch ( Exception e ) {\n      throw new KettleException( \"Error serializing rows of data to the COPY command\", e );\n    }\n\n  }\n\n  protected void verifyDatabaseConnection() throws KettleException {\n    // Confirming Database Connection is defined.\n    if ( meta.getDatabaseMeta() == null ) {\n      throw new KettleException( BaseMessages.getString( PKG, \"PGBulkLoaderMeta.GetSQL.NoConnectionDefined\" ) );\n    }\n  }\n\n  public boolean init( StepMetaInterface smi, StepDataInterface sdi ) {\n    meta = (PGBulkLoaderMeta) smi;\n    data = (PGBulkLoaderData) sdi;\n\n    String enclosure = environmentSubstitute( meta.getEnclosure() );\n    String separator = environmentSubstitute( meta.getDelimiter() );\n","sourceCodeStart":405,"sourceCodeEnd":441,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/plugins/postgresql-db-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/pgbulkloader/PGBulkLoader.java#L405-L441","documentation":"The outer catch of writeRowToPostgres wraps any exception thrown while serializing a single row into the PGCopyOutputStream (field conversion, encoding, stream I/O) into a KettleException with this fixed message. The COPY stream could not accept the serialized row.","triggerScenarios":"Any Exception during row serialization: value formatting errors, bytes not encodable in clientEncoding (CharacterEncodingException), pgCopyOut.write failing due to a broken connection/IOException, or NPEs on null handling paths.","commonSituations":"Client encoding (e.g. UTF-8) mismatch with actual data bytes; network drop mid-load causing stream write failure; null value reaching a write path that dereferences it; malformed numeric data failing formatNumber.","solutions":["Inspect the wrapped cause (e) for the underlying serialization or I/O failure and fix that specific field/encoding","Verify the step's client encoding setting matches the data (e.g. set to UTF-8 and ensure upstream data is valid UTF-8)","Check DB connectivity — a broken pipe mid-COPY surfaces here; reconnect/restart the transformation","Null-guard or default anomalous fields upstream before the loader"],"exampleFix":"// before\npgCopyOut.write( dateTimeString.getBytes( clientEncoding ) ); // throws for unencodable chars\n// after\ntry {\n  pgCopyOut.write( dateTimeString.getBytes( clientEncoding ) );\n} catch ( CharacterEncodingException ce ) {\n  pgCopyOut.write( dateTimeString.getBytes( \"UTF-8\" ) ); // or sanitize input data\n}","handlingStrategy":"try-catch","validationCode":"// verify data is encodable in the configured client encoding before loading\nbyte[] probe = sampleValue.getBytes( java.nio.charset.Charset.forName( clientEncoding ) ); // throws if not encodable","typeGuard":null,"tryCatchPattern":"try {\n  step.processRow();\n} catch ( KettleException e ) {\n  log.error( \"Row serialization failed: \" + e.getMessage(), e.getCause() );\n  // check DB connection health and data encoding, then re-run\n}","preventionTips":["Match the step's client encoding to the actual data encoding (usually UTF-8)","Sanitize or default null/malformed values upstream","Ensure network stability for long COPY loads; increase connection timeouts","Log the failing row number to isolate bad data"],"tags":["postgresql","copy","serialization","io"],"backgroundTag":"database-write-failed","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}