What causes "Unable to flush open files" in pentaho/pentaho-kettle?

Check disk space and filesystem health of the output directory

pentaho/pentaho-kettle · error · KettleException

Unable to flush open files

Error message

Unable to flush open files

What it means

During row processing TextFileOutput periodically flushes all open file streams (every flushInterval). If flushing the FileStreamsCollection raises IOException, it is wrapped in a KettleException 'Unable to flush open files'.

Solutions

Check disk space and filesystem health of the output directory
Retry the transformation after clearing the IO condition
Verify flush interval is reasonable and target storage is writable; check OS-level file handle limits

Defensive patterns

Strategy: retry

Validate before calling

// before run
long usable = new File( outputDir ).getUsableSpace();
if ( usable < minRequiredBytes ) throw new IllegalStateException( "Not enough disk space: " + usable );

Try / catch

try { writeRowToFile( row ); } catch ( KettleException e ) { if ( e.getCause() instanceof IOException && hasFreeSpace() ) { retryFlush(); } else { throw e; } }

Prevention

Monitor disk space on output volumes
Avoid network mounts that drop connections for large runs
Set a sane flush interval; watch open file handle limits

When it happens

Trigger: Periodic flush timer elapsed during writeRowToFile and flushOpenFiles(false) hit an IO error on one of the open files (disk full, I/O error, stream closed).

Common situations: Disk full on the target volume; NFS mount dropped; many files open (split-every rows) and an fd was closed externally.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/02f5ff1a61d85be3. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/steps/textfileoutput/TextFileOutput.java:395

      }

      writeRow( data.outputRowMeta, row );
      putRow( data.outputRowMeta, row ); // in case we want it to go further...

      if ( checkFeedback( getLinesOutput() ) ) {
        logBasic( "linenr " + getLinesOutput() );
      }

      int flushInterval = getFlushInterval();
      if ( flushInterval > 0 ) {
        long currentTime = new Date().getTime();
        if ( data.lastFileFlushTime == 0 ) {
          data.lastFileFlushTime = currentTime;
        } else if ( currentTime - data.lastFileFlushTime > flushInterval ) {
          try {
            data.getFileStreamsCollection().flushOpenFiles( false );
          } catch ( IOException e ) {
            throw new KettleException( "Unable to flush open files", e );
          }
          data.lastFileFlushTime = new Date().getTime();
        }
      }
      return true;
    } else {
      if ( data.writer != null ) {
        if ( data.outputRowMeta != null && meta.isFooterEnabled() ) {
          writeHeader();
        }
      } else if ( !Utils.isEmpty( environmentSubstitute( meta.getEndedLine() ) ) && !meta.isFileNameInField() ) {
        String filename = getOutputFileName( null );
        initFileStreamWriter( filename );
        initBinaryDataFields();
      }
      if ( data.writer != null ) {
        writeEndedLine();
      }

View on GitHub (pinned to f3058517a1)