{"record":{"id":"aa15c503a2e49fea","repo":"infiniflow/ragflow","slug":"bigquery-configured-columns-not-found-in-schema","errorCode":null,"errorMessage":"BigQuery configured columns not found in schema: {', '.join(sorted(missing))}","messagePattern":"BigQuery configured columns not found in schema: (.+?)","errorType":"validation","errorClass":"ConnectorValidationError","httpStatus":null,"severity":"error","filePath":"common/data_source/bigquery_connector.py","lineNumber":631,"sourceCode":"                location=self.location or None,\n            )\n            estimated_bytes = getattr(dry_run_job, \"total_bytes_processed\", None)\n            if estimated_bytes is not None:\n                logging.info(\"BigQuery base query dry-run estimate: %s bytes processed.\", estimated_bytes)\n\n            schema = self._resolve_schema(client, dry_run_job)\n            schema_columns = {field.name for field in schema}\n\n            required = set(self.content_columns)\n            optional = set(self.metadata_columns)\n            if self.id_column:\n                optional.add(self.id_column)\n            if self.timestamp_column:\n                optional.add(self.timestamp_column)\n\n            missing = (required | optional) - schema_columns\n            if missing:\n                raise ConnectorValidationError(f\"BigQuery configured columns not found in schema: {', '.join(sorted(missing))}\")\n\n            if self.timestamp_column:\n                self._resolve_cursor_param_type()\n        except (ConnectorValidationError, ConnectorMissingCredentialError):\n            raise\n        except Exception as exc:\n            raise ConnectorValidationError(f\"BigQuery validation failed: {exc}\")\n","sourceCodeStart":613,"sourceCodeEnd":639,"githubUrl":"https://github.com/infiniflow/ragflow/blob/554fb1133ac3861732235ad9c377eb5e0a770665/common/data_source/bigquery_connector.py#L613-L639","documentation":"After a dry-run resolves the schema, every configured column (content_columns plus optional metadata/id/timestamp columns) must exist in the schema; missing ones are reported as a sorted, comma-joined list.","triggerScenarios":"Any of content_columns, metadata_columns, id_column, or timestamp_column naming a field not produced by the base query or table. The set difference (required | optional) - schema_columns is non-empty.","commonSituations":"Renamed or dropped warehouse columns; custom query aliasing columns to new names; case mismatches; copying a config between tables with different schemas.","solutions":["Match the reported missing names against the table/query schema and fix the config (spelling, case, alias)","If a column is optional metadata you no longer need, remove it from metadata_columns","For custom queries, add the missing columns to the SELECT list"],"exampleFix":"// before\nconfig = {\"content_columns\": [\"bodies\"], ...}  # schema has 'body'\n\n// after\nconfig = {\"content_columns\": [\"body\"], ...}","handlingStrategy":"validation","validationCode":"schema_cols = {f.name for f in client.get_table(f\"{p}.{d}.{t}\").schema}\nmissing = (set(config[\"content_columns\"]) | set(config.get(\"metadata_columns\", []))\n           | ({config[\"id_column\"]} if config.get(\"id_column\") else set())\n           | ({config[\"timestamp_column\"]} if config.get(\"timestamp_column\") else set())) - schema_cols\nif missing:\n    raise ValueError(f\"columns not in schema: {sorted(missing)}\")","typeGuard":null,"tryCatchPattern":"try:\n    conn.validate_connector_settings()\nexcept ConnectorValidationError as e:\n    if \"not found in schema\" in str(e):\n        sync_config_with_actual_schema()  # rename/remove mapped columns\n    else:\n        raise","preventionTips":["Derive column config from the live schema (picker), not free text","Re-validate connectors after warehouse schema migrations","Parse the sorted missing list in the message to auto-remedate column maps"],"tags":["bigquery","schema","configuration","validation"],"backgroundTag":null,"analyzedSha":"554fb1133ac3861732235ad9c377eb5e0a770665","analyzedAt":"2026-08-15T09:20:16.380Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}