What causes "Error writing marker flag" in pentaho/pentaho-kettle?

Check the cause IOException on the KettleFileException (broken pipe vs closed stream) to diagnose the transport.

pentaho/pentaho-kettle · error · org.pentaho.di.core.exception.KettleFileException

Error writing marker flag

Error message

Error writing marker flag

What it means

writeData serializes row data to an OutputStream and first writes a boolean marker flag when the row metadata has zero fields. If outputStream.writeBoolean throws IOException, it is wrapped in a KettleFileException with the message "Error writing marker flag". This signals the underlying stream failed mid-write.

Solutions

Check the cause IOException on the KettleFileException (broken pipe vs closed stream) to diagnose the transport.
Ensure the OutputStream is open and connected before serialization; re-establish the socket/connection and retry the transfer.
For sockets, increase timeouts and validate the peer is still alive before writing large row streams.
Wrap stream lifecycle in try-with-resources so streams are never written after close.

Example fix

// before
rowMeta.writeData(outputStream, rowData); // NPE/ke: Error writing marker flag if stream dead
// after
if (!isStreamOpen(outputStream)) { reconnect(); }
try {
  rowMeta.writeData(outputStream, rowData);
} catch (KettleFileException e) {
  log.error("Failed writing marker flag: " + e.getCause(), e);
  reconnectAndResend(rowData);
}

Defensive patterns

Strategy: retry

Validate before calling

if (outputStream == null || !isStreamWritable(outputStream)) { throw new IllegalStateException("Output stream not writable"); }

Try / catch

try {
  rowMeta.writeData(outputStream, rowData);
  outputStream.flush();
} catch (KettleFileException e) {
  if (e.getCause() instanceof IOException) { reconnectAndRetry(rowData); }
  else throw e;
}

Prevention

Flush and verify connectivity before streaming rows.
Set socket timeouts and keep-alive on clustered transfers.
Never reuse a closed stream; manage lifecycle with try-with-resources.

When it happens

Trigger: Calling RowMeta.writeData(outputStream, data) with size()==0 metadata and the underlying OutputStream throws IOException (socket closed, pipe broken, disk full, stream already closed).

Common situations: Writing row data across a socket where the peer disconnected; writing to a closed ByteArrayOutputStream/FileOutputStream; network interruption in clustered/transformation slave-server streaming.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/cd994454289c7b8b. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/pentaho/di/core/row/RowMeta.java:684

   * @throws KettleFileException in case things go awry
   */
  @Override
  public void writeData( DataOutputStream outputStream, Object[] data ) throws KettleFileException {
    lock.readLock().lock();
    try {
      // Write all values in the row
      for ( int i = 0; i < size(); i++ ) {
        getValueMeta( i ).writeData( outputStream, data[ i ] );
      }

      // If there are 0 values in the row, we write a marker flag to be able to detect an EOF on the other end (sockets
      // etc)
      //
      if ( size() == 0 ) {
        try {
          outputStream.writeBoolean( true );
        } catch ( IOException e ) {
          throw new KettleFileException( "Error writing marker flag", e );
        }
      }
    } finally {
      lock.readLock().unlock();
    }
  }

  /**
   * Write ONLY the specified metadata to the outputStream
   *
   * @throws KettleFileException in case things go awry
   */
  @Override
  public void writeMeta( DataOutputStream outputStream ) throws KettleFileException {
    lock.readLock().lock();
    try {
      // First handle the number of fields in a row
      try {

View on GitHub (pinned to f3058517a1)