{"record":{"id":"6a300fb191f41f81","repo":"prestodb/presto","slug":"bigquery-table-disappear-during-list","errorCode":"BIGQUERY_TABLE_DISAPPEAR_DURING_LIST","errorMessage":"Table disappeared during listing operation","messagePattern":"Table disappeared during listing operation","errorType":"error_code","errorClass":"BigQueryException","httpStatus":null,"severity":"warning","filePath":"presto-bigquery/src/main/java/com/facebook/presto/plugin/bigquery/BigQueryMetadata.java","lineNumber":222,"sourceCode":"            ConnectorTableHandle tableHandle,\n            ColumnHandle columnHandle)\n    {\n        log.debug(\"getColumnMetadata(session=%s, tableHandle=%s, columnHandle=%s)\", session, columnHandle, columnHandle);\n        return ((BigQueryColumnHandle) columnHandle).getColumnMetadata();\n    }\n\n    @Override\n    public Map<SchemaTableName, List<ColumnMetadata>> listTableColumns(ConnectorSession session, SchemaTablePrefix prefix)\n    {\n        log.debug(\"listTableColumns(session=%s, prefix=%s)\", session, prefix);\n        requireNonNull(prefix, \"prefix is null\");\n        ImmutableMap.Builder<SchemaTableName, List<ColumnMetadata>> columns = ImmutableMap.builder();\n        for (SchemaTableName tableName : listTables(session, prefix)) {\n            try {\n                columns.put(tableName, getTableMetadata(session, tableName).getColumns());\n            }\n            catch (NotFoundException ex) {\n                throw new BigQueryException(BIGQUERY_TABLE_DISAPPEAR_DURING_LIST, \"Table disappeared during listing operation\", ex);\n            }\n        }\n        return columns.build();\n    }\n\n    private List<SchemaTableName> listTables(ConnectorSession session, SchemaTablePrefix prefix)\n    {\n        if (prefix.getTableName() == null) {\n            return listTables(session, Optional.ofNullable(prefix.getSchemaName()));\n        }\n        SchemaTableName tableName = prefix.toSchemaTableName();\n        Optional<TableInfo> tableInfo = getBigQueryTable(tableName);\n        return tableInfo.isPresent() ?\n                ImmutableList.of(tableName) :\n                ImmutableList.of(); // table does not exist\n    }\n\n    @Override","sourceCodeStart":204,"sourceCodeEnd":240,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-bigquery/src/main/java/com/facebook/presto/plugin/bigquery/BigQueryMetadata.java#L204-L240","documentation":"Thrown by BigQueryMetadata.listTableColumns when a table that was just returned by listTables no longer exists by the time getTableMetadata fetches its schema, surfacing the BigQuery NotFoundException as BIGQUERY_TABLE_DISAPPEAR_DURING_LIST. It reflects a race between listing tables in a schema and reading each table's metadata: an external actor (or a stale/permission-limited view) removed or hid the table mid-iteration.","triggerScenarios":"During a listing operation with a SchemaTablePrefix (e.g. SHOW COLUMNS / metadata queries over all tables), getTableMetadata(session, tableName) throws com.google.api.gax.httpjson NotFoundException (HTTP 404) for one of the tables returned moments earlier by listTables.","commonSituations":"Concurrent DROP TABLE in the same schema while Presto lists columns; a视图 or partition-time tables being recreated; eventual consistency after dataset/table churn; IAM changes causing a 404-masquerading permission failure; hourly/daily table drops by scheduled jobs.","solutions":["Re-run the listing query; transient races usually succeed on retry","Confirm no concurrent job is dropping/renaming tables in the target dataset during the query","Check IAM permissions on the dataset — missing permissions can surface as 404 NotFound","Pin the listing to specific table names instead of a schema-wide prefix to shrink the race window","If frequent, add retry-on-NotFound handling around listTableColumns in the connector or skip missing tables"],"exampleFix":"// before: whole-schema listing races with DROP TABLE\nSELECT * FROM information_schema.columns WHERE table_schema = 'logs';\n// after: target the specific surviving tables\nSELECT * FROM information_schema.columns WHERE table_schema = 'logs' AND table_name = 'events';","handlingStrategy":"retry","validationCode":"for (SchemaTableName t : expectedTables) {\n    // pre-check existence before expensive listing\n    try (Table t2 = bigQuery.getTable(tableId(t))) {\n        if (t2 == null) skip(t);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    columns = metadata.listTableColumns(session, prefix);\n} catch (BigQueryException e) {\n    if (e.getCode() == BIGQUERY_TABLE_DISAPPEAR_DURING_LIST && attempt < 3) {\n        backoff(attempt); // 100ms * 2^attempt\n        continue;\n    }\n    throw e;\n}","preventionTips":["Avoid schema-wide listings while ETL jobs drop/recreate tables; schedule them apart","Query specific tables via information_schema with table_name filters to narrow the race window","Check dataset IAM — ensure the principal has adequate visibility to avoid 404s","Treat BIGQUERY_TABLE_DISAPPEAR_DURING_LIST as retryable (HTTP 404 race) in orchestration"],"tags":["bigquery","metadata","race-condition","not-found"],"backgroundTag":"table-not-found-during-listing","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}