apache/beam · warning

Could not encode elements from

Error message

Could not encode elements from "{}" to bytes: {}

What it means

DataSampler's handleDataSampleRequest collects sampled elements for a PCollection and must encode them to bytes for the response; if the coder's encoding throws IOException, this warning is logged and that PCollection's samples are omitted from the response. Data sampling continues for other PCollections.

Solutions

1. Fix the custom coder throwing IOException so it can encode the sampled elements.
2. Disable data sampling (--dataSampler) if it is only used for debugging and not needed.
3. Check that sampled elements were not mutated after emission in your DoFn.
4. Capture the full stack trace (enable debug logging) to identify which coder failed.

Example fix

// before
public class BadCoder<T> extends CustomCoder<T> {
  public void encode(T value, OutputStream out) {
    // forgot to handle IOException / serialized object that changed
  }
}
// after
public class GoodCoder<T> extends CustomCoder<T> {
  public void encode(T value, OutputStream out) throws IOException {
    out.write(serialize(value)); // handle the actual bytes safely
  }
}

// Verify your coder round-trips sampled values before enabling data sampling
Coder<T> coder = ...;
ByteArrayOutputStream out = new ByteArrayOutputStream();
coder.encode(sample, out);
assert coder.decode(new ByteArrayInputStream(out.toByteArray())) != null;

try {
  response.putElementSamples(pcollectionId, ...);
} catch (IOException e) {
  LOG.warn("Could not encode elements: {}", e.toString());
}

When it happens

Trigger: A registered data sampler for pcollectionId has samples whose type/ coder cannot be re-encoded when building the SampleData instruction response (coder IOException during samples() materialization).

Common situations: Custom coders that fail on re-encoding sampled values; objects mutated after being sampled so their coder now fails; debugging sessions with data sampling enabled on PCollections with fragile coders.

Related errors

• ApproximateUnique.PerKey requires its input to use KvCoder
• AvroCoder for GenericRecord requires a schema
• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/121266be05479ab8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/debug/DataSampler.java:176

    BeamFnApi.SampleDataRequest sampleDataRequest = request.getSampleData();

    List<String> pcollections = sampleDataRequest.getPcollectionIdsList();

    // Safe to iterate as the ConcurrentHashMap will return each element at most once and will not
    // throw ConcurrentModificationException.
    BeamFnApi.SampleDataResponse.Builder response = BeamFnApi.SampleDataResponse.newBuilder();
    outputSamplers.forEach(
        (pcollectionId, outputSampler) -> {
          if (!pcollections.isEmpty() && !pcollections.contains(pcollectionId)) {
            return;
          }

          try {
            response.putElementSamples(
                pcollectionId,
                ElementList.newBuilder().addAllElements(outputSampler.samples()).build());
          } catch (IOException e) {
            LOG.warn(
                "Could not encode elements from \"{}\" to bytes: {}", pcollectionId, e.toString());
          }
        });

    return BeamFnApi.InstructionResponse.newBuilder().setSampleData(response);
  }
}

View on GitHub (pinned to 12126d8942)