apache/druid · error · IllegalArgumentException

An external S3 table with a format must also provide the cor

Error message

An external S3 table with a format must also provide the corresponding columns

What it means

When defining an external S3 table in the Druid catalog, specifying an input format implies the table will be parsed from structured files, so the schema is required. Druid throws this IllegalArgumentException during table validation when the table spec defines an inputFormat but provides no column definitions. The columns are needed to interpret the formatted data.

Source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/catalog/model/table/S3InputSourceDefn.java:140

  public String typeValue()
  {
    return TYPE_KEY;
  }

  @Override
  protected Class<? extends InputSource> inputSourceClass()
  {
    return S3InputSource.class;
  }

  @Override
  public void validate(ResolvedExternalTable table)
  {
    final boolean hasFormat = table.inputFormatMap != null;
    final boolean hasColumns = !CollectionUtils.isNullOrEmpty(table.resolvedTable().spec().columns());

    if (hasFormat && !hasColumns) {
      throw new IAE(
          "An external S3 table with a format must also provide the corresponding columns"
      );
    }

    // The user can either provide a bucket, or can provide one of the valid items.
    final String bucket = table.resolvedTable().stringProperty(BUCKET_PROPERTY);
    final boolean hasBucket = bucket != null;
    final Map<String, Object> sourceMap = table.inputSourceMap;
    final boolean hasUris = sourceMap.containsKey(URIS_FIELD);
    final boolean hasPrefix = sourceMap.containsKey(PREFIXES_FIELD);
    final boolean hasObjects = sourceMap.containsKey(OBJECTS_FIELD);
    final boolean hasGlob = sourceMap.containsKey(OBJECT_GLOB_FIELD);
    if (hasBucket) {
      if (hasUris || hasPrefix || hasObjects) {
        throw new IAE(
            "Provide either the %s property, or one of the S3 input source fields %s, %s or %s, but not both.",
            BUCKET_PROPERTY,
            URIS_FIELD,

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Add the column definitions matching the input format to the table spec's columns list
2. Alternatively remove the inputFormat if the columns are genuinely unknown and the format is inferred at ingestion time
3. Validate the table spec locally before submitting it to the catalog

Example fix

// before
{"external": {"inputSource": {...}, "inputFormat": {"type": "csv"}}}
// after
{"external": {"inputSource": {...}, "inputFormat": {"type": "csv"}, "columns": [{"name": "ts", "type": "string"}, {"name": "val", "type": "long"}]}}

if (externalTable.inputFormat != null && (externalTable.columns == null || externalTable.columns.isEmpty())) {
  throw new IllegalArgumentException("inputFormat requires column definitions");
}

boolean hasColumns(Map<String, Object> spec) {
  Object cols = spec.get("columns");
  return cols instanceof List && !((List<?>) cols).isEmpty();
}

When it happens

Trigger: Calling validate() on a ResolvedExternalTable whose inputFormatMap is non-null (an input format like JSON/CSV/Parquet is set) while table.resolvedTable().spec().columns() is null or empty.

Common situations: Creating or updating an external S3 table via the catalog API or SQL EXTERNAL DDL with a format (e.g. FORMAT='CSV') but omitting the COLUMNS clause; migrating table specs from a source that supplies format at read time instead of catalog time.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Provide either the %s property, or one of the S3 input sourc
• The %s property cannot be provided when the %s property is s
• S3 external table defines the %s property. The table functio
• S3 requires one of %s, %s or %s
• When using the %s parameter, %s must also be provided

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/826ee5fef9ca2558. Report an issue: GitHub.