pentaho/pentaho-kettle · critical · KettleStepException

A deadlock was detected between steps

Error message

A deadlock was detected between steps '{0}' and '{1}'.  The steps are both waiting for each other because a series of row set buffers filled up.

What it means

During transformation startup, BaseStep detects that two steps that depend on each other have both filled their row set buffers and are waiting for each other — a deadlock in the step graph. KettleStepException is thrown when transMeta.findPrevious confirms the peer step is actually an upstream dependency of the current step.

Solutions

Remove the cyclic hop in the transformation canvas so the step graph is a DAG.
If a feedback loop is intended, break the cycle with a blocking/cacheing step (e.g. Stream Lookup with a cache, or write to a file/DB and read back in a separate transformation).
Split the transformation into two transformations executed in sequence (a Job with two Transformation entries).
Review 'copies' and hop directions in the failing pair reported in the message and confirm which row set filled first.

Example fix

// before (step graph)
A -> B -> A  (cycle)
// after
A -> B ; B -> file ; job step: transformation2 reads file

Defensive patterns

Strategy: try-catch

Validate before calling

// Before running: detect cycles in the step graph
List<StepMeta> hops = transMeta.getSteps();
java.util.Set<StepMeta> visited = new java.util.HashSet<>();
for (StepMeta s : hops) { if (hasCycle(transMeta, s, visited)) throw new IllegalStateException("Cyclic hop detected involving " + s.getName()); }

Try / catch

try { trans.prepareExecution(new Object[0]); trans.startThreads(); trans.waitUntilFinished(); } catch (KettleException e) { if (e.getMessage() != null && e.getMessage().contains("deadlock was detected")) { logError("Cycle between steps; refactor to DAG"); } throw e; }

Prevention

Never draw hops that close a loop between two steps
Use a Job with sequential transformations for feedback patterns
Review hop arrows in Spoon after copy-pasting step groups
Keep 'Block until steps finish' usage minimal and acyclic

When it happens

Trigger: A transformation where step A reads from step B's rowset while B (directly or via a cycle) reads from A, and both row sets are full; detected in the row set fullness audit while steps run single-threaded waiting on getRow/putRow.

Common situations: Accidental circular hop (A->B->A) drawn in Spoon; a feedback loop without a 'Stream Lookup'-style buffering design; 'Block until steps finish' combined with cross-reading between mutually dependent sub-transformations.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of pentaho/pentaho-kettle@f3058517a1 (2026-09-13). Data as JSON: /api/errors/a9546df7ef14de80. Report an issue: GitHub.

Appendix: source

Thrown at engine/src/main/java/org/pentaho/di/trans/step/BaseStep.java:2374

          inputSize += rowSet.size();
        }
        // All full probably means a stalled step.
        List<RowSet> combiOutputRowSets = combi.step.getOutputRowSets();
        if ( inputSize > 0 && inputSize == totalSize && combiOutputRowSets.size() > 1 ) {
          RowSet outputFull = null;
          RowSet outputEmpty = null;
          for ( RowSet rowSet : combiOutputRowSets ) {
            if ( rowSet.size() == transMeta.getSizeRowset() ) {
              outputFull = rowSet;
            } else if ( rowSet.size() == 0 ) {
              outputEmpty = rowSet;
            }
          }
          if ( outputFull != null && outputEmpty != null ) {
            // Verify that this step is lated before the current one
            //
            if ( transMeta.findPrevious( stepMeta, combi.stepMeta ) ) {
              throw new KettleStepException( "A deadlock was detected between steps '"
                + combi.stepname + "' and '" + stepname
                + "'.  The steps are both waiting for each other because a series of row set buffers filled up." );
            }
          }
        }
      }
    }
  }

  /**
   * Find input row set.
   *
   * @param sourceStep the source step
   * @return the row set
   * @throws KettleStepException the kettle step exception
   */
  public RowSet findInputRowSet( String sourceStep ) throws KettleStepException {
    // Check to see that "sourceStep" only runs in a single copy

View on GitHub (pinned to f3058517a1)