prestodb/presto · error · IllegalArgumentException

Unexpected partition: %s. Total number of partitions: %s.

Error message

Unexpected partition: %s. Total number of partitions: %s.

What it means

PrestoSparkPartitioner maps a MutablePartitionId key to its partition number for Spark shuffles. getPartition validates the stored partition is within [0, numPartitions) and throws IllegalArgumentException when the key carries an out-of-range partition value, indicating corrupted or miscomputed partition ids.

Source

Thrown at presto-spark-classloader-interface/src/main/java/com/facebook/presto/spark/classloader_interface/PrestoSparkPartitioner.java:44

    public PrestoSparkPartitioner(int numPartitions)
    {
        this.numPartitions = numPartitions;
    }

    @Override
    public int numPartitions()
    {
        return numPartitions;
    }

    @Override
    public int getPartition(Object key)
    {
        requireNonNull(key, "key is null");
        MutablePartitionId mutablePartitionId = (MutablePartitionId) key;
        int partition = mutablePartitionId.getPartition();
        if (!(partition >= 0 && partition < numPartitions)) {
            throw new IllegalArgumentException(format("Unexpected partition: %s. Total number of partitions: %s.", partition, numPartitions));
        }
        return partition;
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify all partition ids written into MutablePartitionId are 0 <= id < numPartitions
2. Ensure the numPartitions passed to PrestoSparkPartitioner matches the number of output partitions used upstream
3. Check for version skew between nodes producing the shuffle data

Example fix

// before
new MutablePartitionId(-1, position);
// after
int pid = Math.max(0, Math.min(computedPartition, numPartitions - 1));
new MutablePartitionId(pid, position);

if (partitionId < 0 || partitionId >= numPartitions) { throw new IllegalArgumentException("partition out of range: " + partitionId); }

try { int p = partitioner.getPartition(key); } catch (IllegalArgumentException e) { /* inspect/repair MutablePartitionId contents */ throw e; }

When it happens

Trigger: Passing a MutablePartitionId whose getPartition() value is negative or >= numPartitions into a Spark operation (e.g. partitionBy/saveAsNewAPIHadoopDataset) using this partitioner.

Common situations: Bug in upstream partition-id assignment; wrong numPartitions supplied to the partitioner compared with the ids produced; mixed worker versions producing incompatible partition ids during shuffle.

Related errors

• NOT_SUPPORTED
• HUDI_INVALID_PARTITION_VALUE
• NOT_SUPPORTED
• Unexpected partitioning:
• Error creating temporary directory 'local_shuffle'.

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/45fb1b5cf48e91ca. Report an issue: GitHub.