apache/beam · error · IllegalArgumentException

BigQuery data contained value %s with sub-millisecond precis

Error message

BigQuery data contained value %s with sub-millisecond precision, which Beam does not currently support. You can enable truncating timestamps to millisecond precision by using BigQueryIO.withTruncatedTimestamps

What it means

Thrown by safeToMillis when the default REJECT truncation mode finds a BigQuery timestamp Avro value (microseconds since epoch) whose sub-second remainder is not a whole number of milliseconds (value % 1000 != 0). Beam's Instant cannot represent sub-millisecond precision, so in REJECT mode the conversion fails instead of silently losing precision. The message tells you the sanctioned workaround: withTruncatedTimestamps.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:1093

      case ROW:
        Schema rowSchema = beamFieldType.getRowSchema();
        if (rowSchema == null) {
          throw new IllegalArgumentException("Nested ROW missing row schema");
        }
        GenericData.Record record = (GenericData.Record) avroValue;
        return toBeamRow(record, rowSchema, options);
      case MAP:
        return convertAvroRecordToMap(beamFieldType, avroValue, options);
      default:
        throw new RuntimeException(
            "Does not support converting unknown type value: " + beamFieldTypeName);
    }
  }

  private static ReadableInstant safeToMillis(Object value) {
    long subMilliPrecision = ((long) value) % 1000;
    if (subMilliPrecision != 0) {
      throw new IllegalArgumentException(
          String.format(
              "BigQuery data contained value %s with sub-millisecond precision, which Beam does"
                  + " not currently support."
                  + " You can enable truncating timestamps to millisecond precision"
                  + " by using BigQueryIO.withTruncatedTimestamps",
              value));
    } else {
      return truncateToMillis(value);
    }
  }

  private static ReadableInstant truncateToMillis(Object value) {
    return new Instant((long) value / 1000);
  }

  private static Object convertAvroArray(
      FieldType beamField, Object value, BigQueryUtils.ConversionOptions options) {
    // Check whether the type of array element is equal.

View on GitHub (pinned to 12126d8942)

Solutions

Apply BigQueryIO.withTruncatedTimestamps() (or ConversionOptions TRUNCATE mode) so sub-millisecond digits are dropped via truncateToMillis.
Pre-clean the data: round/normalize the BigQuery column to millisecond precision (e.g. TIMESTAMP_TRUNC(col, MILLISECOND)) before export/read.
Catch the IllegalArgumentException around the read conversion and fall back to a truncating conversion for affected records.
If millisecond loss is unacceptable, read the raw microsecond value as INT64/BYTES and construct java.time.Instant yourself.

Example fix

// before
BigQueryIO.readTableRows().fromQuery("SELECT ts FROM t") // REJECT default
// after
BigQueryIO.readTableRows().fromQuery("SELECT ts FROM t").withTruncatedTimestamps()

Defensive patterns

Strategy: try-catch

Validate before calling

// Java: probe data before reading (or pre-round in SQL)
// SELECT TIMESTAMP_TRUNC(ts, MILLISECOND) AS ts FROM t  -- guarantees no sub-ms values

Try / catch

try {
  result = pipeline.apply(BigQueryIO.readTableRows().fromQuery(q));
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("sub-millisecond precision")) {
    // rerun with truncation enabled
    result = pipeline.apply(BigQueryIO.readTableRows().fromQuery(q).withTruncatedTimestamps());
  } else throw e;
}

Prevention

Enable withTruncatedTimestamps() up front when reading TIMESTAMP columns you don't control.
Round timestamps to millisecond precision in SQL (TIMESTAMP_TRUNC) before reading.
Document precision loss when truncation is enabled so downstream consumers know.

When it happens

Trigger: Reading a BigQuery TIMESTAMP column whose stored value has sub-millisecond precision (microsecond timestamps, e.g. '2020-01-01 00:00:00.000123 UTC') via BigQueryIO.readTableRows/read(Schema) with default (REJECT) ConversionOptions.

Common situations: Tables populated by other systems (e.g. Python datetime, other databases) writing microsecond-precision timestamps; BigQuery TIMESTAMP always stores microseconds, so imported data often carries sub-ms digits.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7b6724d81de01983. Report an issue: GitHub.