prestodb/presto · error · PrestoException

HIVE_PARTITION_NOT_FOUND

HIVE_PARTITION_NOT_FOUND

Error message

Failed to fetch partitions after %d retries. %d unprocessed keys remain: %s

What it means

GlueHiveMetastore's batched partition fetch (getPartitionsByNames path) retries GetPartitionsBatch calls when Glue returns unprocessed keys, but after exceeding maxUnprocessedKeysRetries it gives up and throws HIVE_PARTITION_NOT_FOUND with the retry count and the remaining partition values. This means Glue repeatedly failed to return some requested partitions even after retries.

Source

Thrown at presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/glue/GlueHiveMetastore.java:1124

    private List<Partition> batchGetPartition(String databaseName, String tableName, List<String> partitionNames)
    {
        List<CompletableFuture<BatchGetPartitionResponse>> batchGetPartitionFutures = new ArrayList<>();
        try {
            List<PartitionValueList> pendingPartitions = partitionNames.stream()
                    .map(partitionName -> PartitionValueList.builder().values(toPartitionValues(partitionName)).build())
                    .collect(toCollection(ArrayList::new));

            ImmutableList.Builder<Partition> resultsBuilder = ImmutableList.builderWithExpectedSize(partitionNames.size());

            GluePartitionConverter converter = new GluePartitionConverter(databaseName, tableName);

            int retryAttempt = 0;
            while (!pendingPartitions.isEmpty()) {
                // Check if we've exceeded the maximum retry attempts
                if (retryAttempt > 0) {
                    if (retryAttempt > maxUnprocessedKeysRetries) {
                        throw new PrestoException(
                                HIVE_PARTITION_NOT_FOUND,
                                format("Failed to fetch partitions after %d retries. %d unprocessed keys remain: %s",
                                        maxUnprocessedKeysRetries,
                                        pendingPartitions.size(),
                                        pendingPartitions.stream()
                                                .map(p -> p.values().toString())
                                                .limit(10)
                                                .collect(joining(", "))));
                    }

                    long delayMillis = min(
                            unprocessedKeysRetryMinDelayMillis * (1L << (retryAttempt - 1)),
                            unprocessedKeysRetryMaxDelayMillis);

                    log.warn("Retrying %d unprocessed partition keys for table %s.%s (attempt %d/%d) after %dms delay",
                            pendingPartitions.size(), databaseName, tableName, retryAttempt, maxUnprocessedKeysRetries, delayMillis);

                    Thread.sleep(delayMillis);

View on GitHub (pinned to 55bb57d202)

Solutions

Retry the operation once load subsides; the failure is usually transient throttling of GetPartitionsBatch.
Increase maxUnprocessedKeysRetries (hive.metastore.glue max-unprocessed-keys-retries) so more backoff rounds are attempted.
Reduce the batch size / number of partitions requested per call, or partition your query to touch fewer partitions.
Verify the requested partition values actually exist (getPartitionNames) — permanently missing partitions can surface here.

Example fix

// before
List<Partition> parts = metastore.getPartitionsByNames(context, table, manyNames);
// after
// raise retry budget, then split the request
List<Partition> parts = new ArrayList<>();
for (List<String> chunk : partition(manyNames, 100)) {
    parts.addAll(metastore.getPartitionsByNames(context, table, chunk));
}

Defensive patterns

Strategy: retry

Validate before calling

// Confirm the requested partitions exist and bound the request size:
List<String> existing = metastore.getPartitionNames(ctx, table).orElseThrow(...);
List<String> valid = requested.stream().filter(existing::contains).collect(toImmutableList());
// chunk valid into smaller batches before getPartitionsByNames

Try / catch

try {
    return metastore.getPartitionsByNames(ctx, table, names);
}
catch (PrestoException e) {
    if (e.getErrorCode().getName().equals("HIVE_PARTITION_NOT_FOUND")) {
        return retryWithBackoffAfterDelay(names); // throttling usually transient
    }
    throw e;
}

Prevention

Increase hive.metastore.glue max-unprocessed-keys-retries for high-latency environments.
Chunk partition-name requests into smaller batches.
Avoid querying partitions that may be concurrently dropped.
Schedule heavy partition scans outside throttling windows / request quota increases.

When it happens

Trigger: Calling getPartitionsByNames for many specific partition values where Glue's GetPartitionsBatch keeps returning unprocessedKeys across maxUnprocessedKeysRetries attempts — typically sustained throttling, very large batch sizes, or requesting partitions that no longer exist / are being concurrently modified.

Common situations: Queries filtering thousands of partitions during Glue throttling windows; concurrent partition drops/compactions making keys vanish mid-fetch; misconfigured low retry limit (maxUnprocessedKeysRetries) for high-latency environments.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/6c04d8f39ad38e5c. Report an issue: GitHub.