Couldn't set Arrow properties, which may impact read perform

Error message

Couldn't set Arrow properties, which may impact read performance

What it means

A LOG.warn in VectorizedSparkParquetReaders' static initializer: setting Arrow memory-access properties (unsafe memory access, disabling null checks on get) failed via reflection. The vectorized reader still works but with safety checks enabled, degrading read performance. Not fatal — execution continues.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:49

import org.apache.iceberg.spark.SparkUtil;
import org.apache.parquet.schema.MessageType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class VectorizedSparkParquetReaders {

  private static final Logger LOG = LoggerFactory.getLogger(VectorizedSparkParquetReaders.class);
  private static final String ENABLE_UNSAFE_MEMORY_ACCESS = "arrow.enable_unsafe_memory_access";
  private static final String ENABLE_UNSAFE_MEMORY_ACCESS_ENV = "ARROW_ENABLE_UNSAFE_MEMORY_ACCESS";
  private static final String ENABLE_NULL_CHECK_FOR_GET = "arrow.enable_null_check_for_get";
  private static final String ENABLE_NULL_CHECK_FOR_GET_ENV = "ARROW_ENABLE_NULL_CHECK_FOR_GET";

  static {
    try {
      enableUnsafeMemoryAccess();
      disableNullCheckForGet();
    } catch (Exception e) {
      LOG.warn("Couldn't set Arrow properties, which may impact read performance", e);
    }
  }

  private VectorizedSparkParquetReaders() {}

  public static ColumnarBatchReader buildReader(
      Schema expectedSchema,
      MessageType fileSchema,
      Map<Integer, ?> idToConstant,
      BufferAllocator bufferAllocator) {
    return (ColumnarBatchReader)
        TypeWithSchemaVisitor.visit(
            expectedSchema.asStruct(),
            fileSchema,
            new ReaderBuilder(
                expectedSchema,
                fileSchema,
                NullCheckingForGet.NULL_CHECKING_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

Align the iceberg-spark-runtime version with your exact Spark version (e.g. use iceberg-spark-runtime-4.2 for Spark 4.2).
Add JVM flags to permit the needed access if on newer JDKs (e.g. --add-opens java.base/jdk.internal.misc=ALL-UNNAMED, --add-opens java.base/sun.nio.ch=ALL-UNNAMED), mirroring Spark's own flags.
Treat the warning as a performance-only issue: reads still work; investigate only if throughput matters.
Check the chained exception in the log to identify which specific reflective call failed.

Example fix

// before
spark-submit --conf spark.driver.extraJavaOptions=-Xmx4g ...
// after
spark-submit --conf "spark.driver.extraJavaOptions=-Xmx4g --add-opens java.base/jdk.internal.misc=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED"

Defensive patterns

Strategy: validation

Validate before calling

// Ensure Spark/Iceberg/JDK combination is supported before launch
String sparkVersion = spark.version();
if (!supportedIcebergRuntimeFor(sparkVersion)) {
  throw new IllegalStateException("Use iceberg-spark-runtime matching Spark " + sparkVersion);
}

Prevention

Pin iceberg-spark-runtime to the exact Spark major.minor version
Add the same --add-opens JVM flags Spark uses, especially on JDK 17+
Watch driver/executor logs at startup for this warning and fix configuration early
Test vectorized reads after JDK or Spark upgrades

When it happens

Trigger: Class loading of VectorizedSparkParquetReaders when the reflection calls to Arrow/Spark unsafe memory APIs fail — typically due to a Spark/Arrow version mismatch, a JVM without the expected sun.misc.Unsafe access, or a JVM that blocks unsafe operations (e.g. newer JDKs with --illegal-access=deny, or --enable-native-access restrictions).

Common situations: Upgrading Spark or JDK without matching the Iceberg Spark runtime version; running on JDK 16+ where illegal reflective access is denied; custom Arrow builds lacking expected methods.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3a6fd3081e3499ed. Report an issue: GitHub.