prestodb/presto · error · IllegalStateException

Input not found for sourceFragmentId: ${sourceFragmentId}

Error message

Input not found for sourceFragmentId: ${sourceFragmentId}

What it means

PrestoSparkTaskExecutorFactory.fillJavaExecutionTaskInputs resolves each remote source (downstream fragment input) by sourceFragmentId in the provided shuffle/broadcast/in-memory input maps. When a fragment's input has no matching entry, it throws IllegalStateException. This means the task inputs RDD does not cover all remote sources the plan fragment expects.

Source

Thrown at presto-spark-base/src/main/java/com/facebook/presto/spark/execution/task/PrestoSparkTaskExecutorFactory.java:768

                }

                if (broadcastInput != null) {
                    checkArgument(inMemoryInput == null, "single remote source is not expected to accept different kind of inputs");
                    // TODO: Enable NullifyingIterator once migrated to one task per JVM model
                    // NullifyingIterator removes element from the list upon return
                    // This allows GC to gradually reclaim memory
                    // remoteSourcePageInputs.add(getNullifyingIterator(broadcastInput.value()));
                    broadcastInputsListBuilder.add((List<?>) broadcastInput.value());
                    continue;
                }

                if (inMemoryInput != null) {
                    // for in-memory inputs pages can be released incrementally to save memory
                    remoteSourcePageInputsBuilder.add(getNullifyingIterator(inMemoryInput));
                    continue;
                }

                throw new IllegalStateException("Input not found for sourceFragmentId: " + sourceFragmentId);
            }
            List<PrestoSparkShuffleInput> remoteSourceRowInputs = remoteSourceRowInputsBuilder.build();
            List<java.util.Iterator<PrestoSparkSerializedPage>> remoteSourcePageInputs = remoteSourcePageInputsBuilder.build();
            List<List<?>> broadcastInputsList = broadcastInputsListBuilder.build();
            if (!remoteSourceRowInputs.isEmpty()) {
                shuffleInputs.put(remoteSource.getId(), remoteSourceRowInputs);
            }
            if (!remoteSourcePageInputs.isEmpty()) {
                pageInputs.put(remoteSource.getId(), remoteSourcePageInputs);
            }
            if (!broadcastInputsList.isEmpty()) {
                broadcastInputs.put(remoteSource.getId(), broadcastInputsList);
            }
        }
    }

    private List<TaskSource> getTaskSources(Iterator<SerializedPrestoSparkTaskSource> serializedTaskSources)
    {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Ensure coordinator and Spark workers run the same Presto version — mismatched plans cause fragment ID mismatch
2. Check that PrestoSparkRddFactory produced inputs for every remote source of the fragment (inspect rddInputs construction)
3. Clear any cached/stale RDD inputs and re-run the query
4. If using custom input provisioning, verify all sourceFragmentIds from the fragment are populated in the inputs map

Example fix

// before
throw new IllegalStateException("Input not found for sourceFragmentId: " + sourceFragmentId);
// after (fail with diagnostic)
throw new IllegalStateException("Input not found for sourceFragmentId: " + sourceFragmentId
    + ", available fragment ids: " + inputs.getSourceFragmentIds());

Set<String> available = inputs.getSourceFragmentIds();
for (RemoteSourceNode source : fragment.getRemoteSourceNodes()) {
    checkState(available.contains(source.getId().toString()), "missing input for " + source.getId());
}

boolean inputExists(Map<String, ?> inputs, String sourceFragmentId) { return inputs != null && inputs.containsKey(sourceFragmentId); }

try { fillJavaExecutionTaskInputs(...); } catch (IllegalStateException e) { if (e.getMessage().startsWith("Input not found")) { log.error("fragment id mismatch or missing rdd input", e); } throw e; }

When it happens

Trigger: doCreate -> fillJavaExecutionTaskInputs is called with PrestoSparkTaskInputs whose inputs map lacks an entry for a sourceFragmentId referenced by the fragment's RemoteSourceNodes (and the input is not an in-memory input either).

Common situations: Coordinator/executor version skew so fragment IDs differ; custom RDD factory or input providers missing an input; a query plan change that added a remote source not covered by cached RDDs.

Related errors

• Response does not contain a JSON value
• path is empty
• Invalid field position selection after nulls removed: " + se
• Map key is null at position: " + position
• Current entry must be closed before a null can be written

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/0e4886fb19794e65. Report an issue: GitHub.