{"record":{"id":"fffaf1a17572fcda","repo":"pentaho/pentaho-kettle","slug":"error-processing-temp-file","errorCode":null,"errorMessage":"Error processing temp-file!","messagePattern":"Error processing temp-file!","errorType":"exception","errorClass":"KettleException","httpStatus":null,"severity":"error","filePath":"engine/src/main/java/org/pentaho/di/trans/steps/sort/SortRows.java","lineNumber":220,"sourceCode":"      // Close temp-file\n      dos.close(); // close data stream\n      if ( gzos != null ) {\n        gzos.close(); // close gzip stream\n      }\n      outputStream.close(); // close file stream\n\n      // How much memory do we have left?\n      //\n      data.freeMemoryPct = Const.getPercentageFreeMemory();\n      data.freeCounter = 0;\n      if ( data.sortSize <= 0 ) {\n        if ( log.isDetailed() ) {\n          logDetailed( BaseMessages.getString( PKG, \"SortRows.Detailed.AvailableMemory\", data.freeMemoryPct ) );\n        }\n      }\n\n    } catch ( Exception e ) {\n      throw new KettleException( \"Error processing temp-file!\", e );\n    }\n\n    data.getBufferIndex = 0;\n  }\n\n  private DataInputStream getDataInputStream( GZIPInputStream gzipInputStream ) {\n    DataInputStream result = new DataInputStream( gzipInputStream );\n    data.gzis.add( gzipInputStream );\n    return result;\n  }\n\n  // get sorted rows from available files in iterative manner.\n  // that means call to this method will continue to return rows\n  // till all temp files will not be read to the end.\n  Object[] getBuffer() throws KettleValueException {\n    Object[] retval;\n\n    // Open all files at once and read one row from each file...","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/pentaho/pentaho-kettle/blob/f3058517a153da500bf4551f46d79b91bf8ec552/engine/src/main/java/org/pentaho/di/trans/steps/sort/SortRows.java#L202-L238","documentation":"SortRows.sortExternalRows() streams the sorted rows back from the temporary GZIP file when the dataset does not fit in memory. Any exception while reading/processing that temp file (missing file, IO error, deserialization failure) is wrapped in a KettleException with this message.","triggerScenarios":"Reading back the sort's temporary file after external sort fails: temp file deleted mid-run (tmpwatch/cleanup), disk full or IO error, %%java.io.tmpdir%% pointing to an unwritable/full directory, or row metadata mismatch causing deserialization errors.","commonSituations":"Large sorts spilling to disk in environments where /tmp is small (small tmpfs) or periodically cleaned; concurrent transformations colliding on temp files; disk quota exceeded.","solutions":["Point the step's 'Temp directory' setting to a large, stable, writable directory instead of the default %%java.io.tmpdir%%.","Check free disk space on the temp filesystem and increase it if the sort volume exceeds capacity.","Ensure no OS cleanup job (tmpwatch/systemd-tmpfiles) removes files during the run; use a dedicated directory.","Re-run with detailed logging (SortRows.Detailed.AvailableMemory) to see memory pressure and reduce 'Sort size' to control spilling.","Verify the transformation is not modified/restarted concurrently sharing the same temp directory."],"exampleFix":"// before\n// Sort step: directory = %%java.io.tmpdir%% (small tmpfs)\n// after\n// directory = /data/etl/tmp (large persistent volume)","handlingStrategy":"fallback","validationCode":"File tmp = new File(System.getProperty(\"java.io.tmpdir\"));\nif (tmp.getUsableSpace() < requiredBytes) { throw new IllegalStateException(\"Insufficient temp space for external sort\"); }","typeGuard":null,"tryCatchPattern":"try { sortStep.execute(); } catch (KettleException e) {\n  if (e.getMessage().contains(\"Error processing temp-file\")) {\n    // redirect temp dir and/or reduce sort size, then retry\n  }\n}","preventionTips":["Configure a dedicated large temp directory in the Sort step instead of /tmp.","Monitor disk space on the temp volume for large sorts.","Exclude the temp directory from OS cleanup jobs during run windows.","Tune 'Sort size' to balance memory vs spilling."],"tags":["kettle","sort","temp-file","io"],"backgroundTag":"file-read-failed","analyzedSha":"f3058517a153da500bf4551f46d79b91bf8ec552","analyzedAt":"2026-09-13T14:04:16.340Z","contentChangedAt":"2026-09-13T14:04:16.340Z","schemaVersion":2},"datasetVersion":"2026-09-20T23:17:15.980Z"}