What causes "Failed to write StreamingOffset to json" in apache/iceberg?

Retry the streaming batch; the failure is typically transient

apache/iceberg · error · UncheckedIOException

Failed to write StreamingOffset to json

Error message

Failed to write StreamingOffset to json

What it means

StreamingOffset.json() serializes the offset to JSON with a JsonGenerator and throws UncheckedIOException with this message if an IOException occurs during generation. Since generation is in-memory over a StringWriter this is rare and usually indicates a JVM/Jackson-level problem.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/StreamingOffset.java:99

    }

    return fromJsonNode(node);
  }

  @Override
  public String json() {
    StringWriter writer = new StringWriter();
    try (JsonGenerator generator = JsonUtil.factory().createGenerator(writer)) {
      generator.writeStartObject();
      generator.writeNumberField(VERSION, CURR_VERSION);
      generator.writeNumberField(SNAPSHOT_ID, snapshotId);
      generator.writeNumberField(POSITION, position);
      generator.writeBooleanField(SCAN_ALL_FILES, scanAllFiles);
      generator.writeEndObject();
      generator.flush();

    } catch (IOException e) {
      throw new UncheckedIOException("Failed to write StreamingOffset to json", e);
    }

    return writer.toString();
  }

  long snapshotId() {
    return snapshotId;
  }

  long position() {
    return position;
  }

  boolean shouldScanAllFiles() {
    return scanAllFiles;
  }

  @Override

View on GitHub (pinned to 86d9c8fc54)

Solutions

Retry the streaming batch; the failure is typically transient
If reproducible, report the full stack trace (wrapped IOException cause) to the Iceberg project
Confirm the Iceberg and Jackson versions are compatible and not mixed on the classpath

Example fix

null

Defensive patterns

Strategy: try-catch

Try / catch

try {
  String json = offset.json();
} catch (UncheckedIOException e) {
  // retry the batch; escalate if reproducible
}

Prevention

Keep Jackson on the classpath at the version Iceberg expects
Avoid mixing Jackson versions in shaded/fat jars
Retry streaming batches on transient serialization failures

When it happens

Trigger: Calling json() to persist a StreamingOffset into Spark's checkpoint while the underlying JsonGenerator/StringWriter throws an IOException (very rare in-memory failure).

Common situations: Rare — seen during checkpoint writes when serialization machinery misbehaves; effectively an internal error wrapper rather than a user-fixable condition.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/a07b3dc467a7ca99. Report an issue: GitHub.