What causes "Error serializing rows of data to the COPY command" in pentaho/pentaho-kettle?

Inspect the wrapped cause (e) for the underlying serialization or I/O failure and fix that specific field/encoding

pentaho/pentaho-kettle · error · KettleException

Error serializing rows of data to the COPY command

Error message

Error serializing rows of data to the COPY command

What it means

The outer catch of writeRowToPostgres wraps any exception thrown while serializing a single row into the PGCopyOutputStream (field conversion, encoding, stream I/O) into a KettleException with this fixed message. The COPY stream could not accept the serialized row.

Solutions

Inspect the wrapped cause (e) for the underlying serialization or I/O failure and fix that specific field/encoding
Verify the step's client encoding setting matches the data (e.g. set to UTF-8 and ensure upstream data is valid UTF-8)
Check DB connectivity — a broken pipe mid-COPY surfaces here; reconnect/restart the transformation
Null-guard or default anomalous fields upstream before the loader

Example fix

// before
pgCopyOut.write( dateTimeString.getBytes( clientEncoding ) ); // throws for unencodable chars
// after
try {
  pgCopyOut.write( dateTimeString.getBytes( clientEncoding ) );
} catch ( CharacterEncodingException ce ) {
  pgCopyOut.write( dateTimeString.getBytes( "UTF-8" ) ); // or sanitize input data
}

Defensive patterns

Strategy: try-catch

Validate before calling

// verify data is encodable in the configured client encoding before loading
byte[] probe = sampleValue.getBytes( java.nio.charset.Charset.forName( clientEncoding ) ); // throws if not encodable

Try / catch

try {
  step.processRow();
} catch ( KettleException e ) {
  log.error( "Row serialization failed: " + e.getMessage(), e.getCause() );
  // check DB connection health and data encoding, then re-run
}

Prevention

Match the step's client encoding to the actual data encoding (usually UTF-8)
Sanitize or default null/malformed values upstream
Ensure network stability for long COPY loads; increase connection timeouts
Log the failing row number to isolate bad data

When it happens

Trigger: Any Exception during row serialization: value formatting errors, bytes not encodable in clientEncoding (CharacterEncodingException), pgCopyOut.write failing due to a broken connection/IOException, or NPEs on null handling paths.

Common situations: Client encoding (e.g. UTF-8) mismatch with actual data bytes; network drop mid-load causing stream write failure; null value reaching a write path that dereferences it; malformed numeric data failing formatNumber.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/e9c5cce072b29fc9. Report an issue: GitHub.

Appendix: source

Thrown at plugins/postgresql-db-bulk-loader/impl/src/main/java/org/pentaho/di/trans/steps/pgbulkloader/PGBulkLoader.java:423

                pgCopyOut.write( (byte[]) valueData );
              } else {
                BigDecimal big = valueMeta.getBigNumber( valueData );
                if ( big != null ) {
                  pgCopyOut.write( big.toString().getBytes( clientEncoding ) );
                }
              }
              break;
            default:
              throw new KettleException( "PGBulkLoader doesn't handle the type " + valueMeta.getTypeDesc() );
          }
        }
      }

      // Now write a newline
      //
      pgCopyOut.write( data.newline );
    } catch ( Exception e ) {
      throw new KettleException( "Error serializing rows of data to the COPY command", e );
    }

  }

  protected void verifyDatabaseConnection() throws KettleException {
    // Confirming Database Connection is defined.
    if ( meta.getDatabaseMeta() == null ) {
      throw new KettleException( BaseMessages.getString( PKG, "PGBulkLoaderMeta.GetSQL.NoConnectionDefined" ) );
    }
  }

  public boolean init( StepMetaInterface smi, StepDataInterface sdi ) {
    meta = (PGBulkLoaderMeta) smi;
    data = (PGBulkLoaderData) sdi;

    String enclosure = environmentSubstitute( meta.getEnclosure() );
    String separator = environmentSubstitute( meta.getDelimiter() );

View on GitHub (pinned to f3058517a1)