{"record":{"id":"826ee5fef9ca2558","repo":"apache/druid","slug":"an-external-s3-table-with-a-format-must-also-provi","errorCode":null,"errorMessage":"An external S3 table with a format must also provide the corresponding columns","messagePattern":"An external S3 table with a format must also provide the corresponding columns","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java","lineNumber":140,"sourceCode":"  public String typeValue()\n  {\n    return TYPE_KEY;\n  }\n\n  @Override\n  protected Class<? extends InputSource> inputSourceClass()\n  {\n    return S3InputSource.class;\n  }\n\n  @Override\n  public void validate(ResolvedExternalTable table)\n  {\n    final boolean hasFormat = table.inputFormatMap != null;\n    final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());\n\n    if (hasFormat && !hasColumns) {\n      throw new IAE(\n          \"An external S3 table with a format must also provide the corresponding columns\"\n      );\n    }\n\n    // The user can either provide a bucket, or can provide one of the valid items.\n    final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);\n    final boolean hasBucket = bucket != null;\n    final Map<String, Object> sourceMap = table.inputSourceMap;\n    final boolean hasUris = sourceMap.containsKey(URIS_FIELD);\n    final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);\n    final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);\n    final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);\n    if (hasBucket) {\n      if (hasUris || hasPrefix || hasObjects) {\n        throw new IAE(\n            \"Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.\",\n            BUCKET_PROPERTY,\n            URIS_FIELD,","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java#L122-L158","documentation":"When defining an external S3 table in the Druid catalog, specifying an input format implies the table will be parsed from structured files, so the schema is required. Druid throws this IllegalArgumentException during table validation when the table spec defines an inputFormat but provides no column definitions. The columns are needed to interpret the formatted data.","triggerScenarios":"Calling validate() on a ResolvedExternalTable whose inputFormatMap is non-null (an input format like JSON/CSV/Parquet is set) while table.resolvedTable().spec().columns() is null or empty.","commonSituations":"Creating or updating an external S3 table via the catalog API or SQL EXTERNAL DDL with a format (e.g. FORMAT='CSV') but omitting the COLUMNS clause; migrating table specs from a source that supplies format at read time instead of catalog time.","solutions":["Add the column definitions matching the input format to the table spec's columns list","Alternatively remove the inputFormat if the columns are genuinely unknown and the format is inferred at ingestion time","Validate the table spec locally before submitting it to the catalog"],"exampleFix":"// before\n{\"external\": {\"inputSource\": {...}, \"inputFormat\": {\"type\": \"csv\"}}}\n// after\n{\"external\": {\"inputSource\": {...}, \"inputFormat\": {\"type\": \"csv\"}, \"columns\": [{\"name\": \"ts\", \"type\": \"string\"}, {\"name\": \"val\", \"type\": \"long\"}]}}","handlingStrategy":"validation","validationCode":"if (externalTable.inputFormat != null && (externalTable.columns == null || externalTable.columns.isEmpty())) {\n  throw new IllegalArgumentException(\"inputFormat requires column definitions\");\n}","typeGuard":"boolean hasColumns(Map<String, Object> spec) {\n  Object cols = spec.get(\"columns\");\n  return cols instanceof List && !((List<?>) cols).isEmpty();\n}","tryCatchPattern":null,"preventionTips":["Always define columns together with inputFormat in external table specs","Lint table-spec JSON before submitting to the catalog","Copy complete specs (format + columns) from working examples"],"tags":["s3","catalog","validation","missing-columns"],"backgroundTag":"missing-required-config-field","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}