{"record":{"id":"d2a61d0fe05af6e2","repo":"prestodb/presto","slug":"hive-partition-dropped-during-query","errorCode":"HIVE_PARTITION_DROPPED_DURING_QUERY","errorMessage":"Partition no longer exists: %s.%s/%s","messagePattern":"Partition no longer exists: (.+?)\\.(.+?)/(.+?)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/HiveSplitManager.java","lineNumber":846,"sourceCode":"                tableName.getTableName(),\n                Lists.transform(partitionBatch, HivePartition::getPartitionId));\n        Map<String, PartitionStatistics> partitionStatistics = ImmutableMap.of();\n        if (domains.isPresent() && isPartitionStatisticsBasedOptimizationEnabled(session)) {\n            partitionStatistics = metastore.getPartitionStatistics(\n                    metastoreContext,\n                    tableName.getSchemaName(),\n                    tableName.getTableName(),\n                    partitionBatch.stream()\n                            .map(hivePartition -> hivePartition.getPartitionId().getPartitionName())\n                            .collect(toImmutableSet()));\n        }\n\n        Map<String, String> partitionNameToLocation = new HashMap<>();\n        ImmutableMap.Builder<String, PartitionSplitInfo> partitionSplitInfoBuilder = ImmutableMap.builder();\n        for (Map.Entry<String, Optional<Partition>> entry : partitions.entrySet()) {\n            ImmutableSet.Builder<ColumnHandle> redundantColumnDomainsBuilder = ImmutableSet.builder();\n            if (!entry.getValue().isPresent()) {\n                throw new PrestoException(HIVE_PARTITION_DROPPED_DURING_QUERY, format(\"Partition no longer exists: %s.%s/%s\", tableName.getSchemaName(), tableName.getTableName(), entry.getKey()));\n            }\n            boolean pruned = false;\n            if (partitionStatistics.containsKey(entry.getKey())) {\n                Map<String, HiveColumnStatistics> columnStatistics = partitionStatistics.get(entry.getKey()).getColumnStatistics();\n                for (Map.Entry<String, HiveColumnHandle> predicateColumnEntry : predicateColumns.entrySet()) {\n                    if (columnStatistics.containsKey(predicateColumnEntry.getKey())) {\n                        Optional<ValueSet> columnsStatisticsValueSet = getColumnStatisticsValueSet(columnStatistics.get(predicateColumnEntry.getKey()), predicateColumnEntry.getValue().getHiveType());\n                        Subfield subfield = new Subfield(predicateColumnEntry.getKey());\n                        if (columnsStatisticsValueSet.isPresent() && domains.get().containsKey(subfield)) {\n                            ValueSet columnPredicateValueSet = domains.get().get(subfield).getValues();\n                            if (!columnPredicateValueSet.overlaps(columnsStatisticsValueSet.get())) {\n                                pruned = true;\n                                break;\n                            }\n                            if (columnPredicateValueSet.contains(columnsStatisticsValueSet.get())) {\n                                redundantColumnDomainsBuilder.add(predicateColumnEntry.getValue());\n                            }\n                        }","sourceCodeStart":828,"sourceCodeEnd":864,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/HiveSplitManager.java#L828-L864","documentation":"Thrown by HiveSplitManager when a partition that existed when the query plan was created no longer exists in the metastore when splits are generated. Presto throws it to fail the query immediately rather than silently skipping the missing partition, since results would otherwise be inconsistent with what the planner expected.","triggerScenarios":"getPartitions produced an Optional.empty() entry for a partition during split generation — i.e., the partition was dropped (ALTER TABLE DROP PARTITION, DROP TABLE, partition overwrite) by another process between query planning and split enumeration.","commonSituations":"Concurrent ETL jobs dropping/overwriting partitions while a long-running analytical query scans the table; streaming ingestion pipelines that purge partitions; retention jobs deleting old partitions during queries on those partitions.","solutions":["Re-run the query; it will pick up the current partition list.","Coordinate drops with queries: pause partition-retention/DROP PARTITION jobs while analytical queries run, or schedule them at off-peak times.","Use HIVE dynamic partition overwrite or insert-only patterns instead of drop-and-recreate for partitions being queried.","Serialize query scheduling and partition maintenance (e.g., via a workflow orchestrator) so they don't overlap on the same table."],"exampleFix":"// before: retention job drops partitions during queries\nALTER TABLE events DROP PARTITION (ds='2026-08-01');\n// after: only drop partitions not seen by running queries, or delay\nALTER TABLE events DROP IF EXISTS PARTITION (ds < '2026-08-01'); -- run after query window","handlingStrategy":"retry","validationCode":"// before running a long query against hot partitions, confirm they still exist\nfor (String part : partitionsToScan) {\n    Partition p = metastoreClient.getPartition(db, table, part); // throws NoSuchObjectException if dropped\n    if (p == null || p.getSd().getLocation() == null) { /* reschedule query or abort */ }\n}","typeGuard":"boolean partitionExists(Optional<Partition> p) { return p != null && p.isPresent(); }","tryCatchPattern":"try { runQuery(sql); } catch (PrestoException e) {\n    if (HIVE_PARTITION_DROPPED_DURING_QUERY.toErrorCode().getCode() == e.getErrorCode().getCode()) {\n        retryQueryWithFreshPlan(sql); // re-plan picks up current partitions\n    } else { throw e; }\n}","preventionTips":["Schedule DROP PARTITION/retention jobs outside analytical query windows","Prefer INSERT OVERWRITE over drop-and-recreate for partitions being queried","Use snapshotting/iceberg-style tables if concurrent mutations are frequent","Monitor query failures of this code to detect interfering jobs"],"tags":["hive","partition","concurrency","metastore"],"backgroundTag":"partition-dropped-during-query","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}