{"record":{"id":"6c04d8f39ad38e5c","repo":"prestodb/presto","slug":"hive-partition-not-found","errorCode":"HIVE_PARTITION_NOT_FOUND","errorMessage":"Failed to fetch partitions after %d retries. %d unprocessed keys remain: %s","messagePattern":"Failed to fetch partitions after (.+?) retries\\. (.+?) unprocessed keys remain: (.+?)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/glue/GlueHiveMetastore.java","lineNumber":1124,"sourceCode":"\n    private List<Partition> batchGetPartition(String databaseName, String tableName, List<String> partitionNames)\n    {\n        List<CompletableFuture<BatchGetPartitionResponse>> batchGetPartitionFutures = new ArrayList<>();\n        try {\n            List<PartitionValueList> pendingPartitions = partitionNames.stream()\n                    .map(partitionName -> PartitionValueList.builder().values(toPartitionValues(partitionName)).build())\n                    .collect(toCollection(ArrayList::new));\n\n            ImmutableList.Builder<Partition> resultsBuilder = ImmutableList.builderWithExpectedSize(partitionNames.size());\n\n            GluePartitionConverter converter = new GluePartitionConverter(databaseName, tableName);\n\n            int retryAttempt = 0;\n            while (!pendingPartitions.isEmpty()) {\n                // Check if we've exceeded the maximum retry attempts\n                if (retryAttempt > 0) {\n                    if (retryAttempt > maxUnprocessedKeysRetries) {\n                        throw new PrestoException(\n                                HIVE_PARTITION_NOT_FOUND,\n                                format(\"Failed to fetch partitions after %d retries. %d unprocessed keys remain: %s\",\n                                        maxUnprocessedKeysRetries,\n                                        pendingPartitions.size(),\n                                        pendingPartitions.stream()\n                                                .map(p -> p.values().toString())\n                                                .limit(10)\n                                                .collect(joining(\", \"))));\n                    }\n\n                    long delayMillis = min(\n                            unprocessedKeysRetryMinDelayMillis * (1L << (retryAttempt - 1)),\n                            unprocessedKeysRetryMaxDelayMillis);\n\n                    log.warn(\"Retrying %d unprocessed partition keys for table %s.%s (attempt %d/%d) after %dms delay\",\n                            pendingPartitions.size(), databaseName, tableName, retryAttempt, maxUnprocessedKeysRetries, delayMillis);\n\n                    Thread.sleep(delayMillis);","sourceCodeStart":1106,"sourceCodeEnd":1142,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/glue/GlueHiveMetastore.java#L1106-L1142","documentation":"GlueHiveMetastore's batched partition fetch (getPartitionsByNames path) retries GetPartitionsBatch calls when Glue returns unprocessed keys, but after exceeding maxUnprocessedKeysRetries it gives up and throws HIVE_PARTITION_NOT_FOUND with the retry count and the remaining partition values. This means Glue repeatedly failed to return some requested partitions even after retries.","triggerScenarios":"Calling getPartitionsByNames for many specific partition values where Glue's GetPartitionsBatch keeps returning unprocessedKeys across maxUnprocessedKeysRetries attempts — typically sustained throttling, very large batch sizes, or requesting partitions that no longer exist / are being concurrently modified.","commonSituations":"Queries filtering thousands of partitions during Glue throttling windows; concurrent partition drops/compactions making keys vanish mid-fetch; misconfigured low retry limit (maxUnprocessedKeysRetries) for high-latency environments.","solutions":["Retry the operation once load subsides; the failure is usually transient throttling of GetPartitionsBatch.","Increase maxUnprocessedKeysRetries (hive.metastore.glue max-unprocessed-keys-retries) so more backoff rounds are attempted.","Reduce the batch size / number of partitions requested per call, or partition your query to touch fewer partitions.","Verify the requested partition values actually exist (getPartitionNames) — permanently missing partitions can surface here."],"exampleFix":"// before\nList<Partition> parts = metastore.getPartitionsByNames(context, table, manyNames);\n// after\n// raise retry budget, then split the request\nList<Partition> parts = new ArrayList<>();\nfor (List<String> chunk : partition(manyNames, 100)) {\n    parts.addAll(metastore.getPartitionsByNames(context, table, chunk));\n}","handlingStrategy":"retry","validationCode":"// Confirm the requested partitions exist and bound the request size:\nList<String> existing = metastore.getPartitionNames(ctx, table).orElseThrow(...);\nList<String> valid = requested.stream().filter(existing::contains).collect(toImmutableList());\n// chunk valid into smaller batches before getPartitionsByNames","typeGuard":null,"tryCatchPattern":"try {\n    return metastore.getPartitionsByNames(ctx, table, names);\n}\ncatch (PrestoException e) {\n    if (e.getErrorCode().getName().equals(\"HIVE_PARTITION_NOT_FOUND\")) {\n        return retryWithBackoffAfterDelay(names); // throttling usually transient\n    }\n    throw e;\n}","preventionTips":["Increase hive.metastore.glue max-unprocessed-keys-retries for high-latency environments.","Chunk partition-name requests into smaller batches.","Avoid querying partitions that may be concurrently dropped.","Schedule heavy partition scans outside throttling windows / request quota increases."],"tags":["glue","partitions","retries","throttling","aws"],"backgroundTag":"aws-throttling","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}