pentaho/pentaho-kettle · error · KettleException

Error processing temp-file!

Error message

Error processing temp-file!

What it means

SortRows.sortExternalRows() streams the sorted rows back from the temporary GZIP file when the dataset does not fit in memory. Any exception while reading/processing that temp file (missing file, IO error, deserialization failure) is wrapped in a KettleException with this message.

Solutions

1. Point the step's 'Temp directory' setting to a large, stable, writable directory instead of the default %%java.io.tmpdir%%.
2. Check free disk space on the temp filesystem and increase it if the sort volume exceeds capacity.
3. Ensure no OS cleanup job (tmpwatch/systemd-tmpfiles) removes files during the run; use a dedicated directory.
4. Re-run with detailed logging (SortRows.Detailed.AvailableMemory) to see memory pressure and reduce 'Sort size' to control spilling.
5. Verify the transformation is not modified/restarted concurrently sharing the same temp directory.

Example fix

// before
// Sort step: directory = %%java.io.tmpdir%% (small tmpfs)
// after
// directory = /data/etl/tmp (large persistent volume)

File tmp = new File(System.getProperty("java.io.tmpdir"));
if (tmp.getUsableSpace() < requiredBytes) { throw new IllegalStateException("Insufficient temp space for external sort"); }

try { sortStep.execute(); } catch (KettleException e) {
  if (e.getMessage().contains("Error processing temp-file")) {
    // redirect temp dir and/or reduce sort size, then retry
  }
}

When it happens

Trigger: Reading back the sort's temporary file after external sort fails: temp file deleted mid-run (tmpwatch/cleanup), disk full or IO error, %%java.io.tmpdir%% pointing to an unwritable/full directory, or row metadata mismatch causing deserialization errors.

Common situations: Large sorts spilling to disk in environments where /tmp is small (small tmpfs) or periodically cleaned; concurrent transformations colliding on temp files; disk quota exceeded.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• GroupBy.Exception.UnableToCreateTemporaryFile
• Cannot open control file
• Cannot open data file
• ChangeFileEncoding.Error.CreatingFile
• Could not read file

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/fffaf1a17572fcda. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/steps/sort/SortRows.java:220

      // Close temp-file
      dos.close(); // close data stream
      if ( gzos != null ) {
        gzos.close(); // close gzip stream
      }
      outputStream.close(); // close file stream

      // How much memory do we have left?
      //
      data.freeMemoryPct = Const.getPercentageFreeMemory();
      data.freeCounter = 0;
      if ( data.sortSize <= 0 ) {
        if ( log.isDetailed() ) {
          logDetailed( BaseMessages.getString( PKG, "SortRows.Detailed.AvailableMemory", data.freeMemoryPct ) );
        }
      }

    } catch ( Exception e ) {
      throw new KettleException( "Error processing temp-file!", e );
    }

    data.getBufferIndex = 0;
  }

  private DataInputStream getDataInputStream( GZIPInputStream gzipInputStream ) {
    DataInputStream result = new DataInputStream( gzipInputStream );
    data.gzis.add( gzipInputStream );
    return result;
  }

  // get sorted rows from available files in iterative manner.
  // that means call to this method will continue to return rows
  // till all temp files will not be read to the end.
  Object[] getBuffer() throws KettleValueException {
    Object[] retval;

    // Open all files at once and read one row from each file...

View on GitHub (pinned to f3058517a1)