Couldn't set Arrow properties, which may impact read perform

Error message

Couldn't set Arrow properties, which may impact read performance

What it means

VectorizedSparkParquetReaders' static initializer sets JVM-level Arrow memory and null-check properties needed for fast vectorized reads. If enabling them throws (e.g. unsupported Arrow/JDK version), the class logs this warning and falls back to safe defaults; reads still work but slower.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java:49

import org.apache.iceberg.spark.SparkUtil;
import org.apache.parquet.schema.MessageType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class VectorizedSparkParquetReaders {

  private static final Logger LOG = LoggerFactory.getLogger(VectorizedSparkParquetReaders.class);
  private static final String ENABLE_UNSAFE_MEMORY_ACCESS = "arrow.enable_unsafe_memory_access";
  private static final String ENABLE_UNSAFE_MEMORY_ACCESS_ENV = "ARROW_ENABLE_UNSAFE_MEMORY_ACCESS";
  private static final String ENABLE_NULL_CHECK_FOR_GET = "arrow.enable_null_check_for_get";
  private static final String ENABLE_NULL_CHECK_FOR_GET_ENV = "ARROW_ENABLE_NULL_CHECK_FOR_GET";

  static {
    try {
      enableUnsafeMemoryAccess();
      disableNullCheckForGet();
    } catch (Exception e) {
      LOG.warn("Couldn't set Arrow properties, which may impact read performance", e);
    }
  }

  private VectorizedSparkParquetReaders() {}

  public static ColumnarBatchReader buildReader(
      Schema expectedSchema,
      MessageType fileSchema,
      Map<Integer, ?> idToConstant,
      BufferAllocator bufferAllocator) {
    return (ColumnarBatchReader)
        TypeWithSchemaVisitor.visit(
            expectedSchema.asStruct(),
            fileSchema,
            new ReaderBuilder(
                expectedSchema,
                fileSchema,
                NullCheckingForGet.NULL_CHECKING_ENABLED,

View on GitHub (pinned to 86d9c8fc54)

Solutions

Align the Arrow version with the one required by the Iceberg Spark runtime (check icebergs spark build for the Arrow version)
Avoid bundling conflicting Arrow jars in the user classpath; let Spark's Iceberg runtime provide it
Verify on a supported JDK; performance warning is non-fatal, so proceed if latency is acceptable

Defensive patterns

Strategy: fallback

Validate before calling

// detect Arrow conflicts before enabling vectorized reads
try (var is = getClass().getResourceAsStream("/META-INF/maven/org.apache.arrow/arrow-memory-core/pom.properties")) {
  Properties p = new Properties(); p.load(is);
  if (!p.getProperty("version").equals(expectedArrowVersion)) LOG.warn("Arrow version mismatch");
}

Try / catch

try {
  runVectorizedQuery(spark);
} catch (Exception e) {
  LOG.warn("Vectorized read degraded; falling back to non-vectorized or row-based reader", e);
}

Prevention

Keep a single Arrow version on the classpath matching the Iceberg Spark runtime
Avoid shading/relocating Arrow independently
Test on the target JDK version for unsafe-memory access
Accept the warning as a performance-only degradation

When it happens

Trigger: Class-loading VectorizedSparkParquetReaders when the static enableUnsafeMemoryAccess()/disableNullCheckForGet() reflection calls fail — typically Arrow version incompatibility or JDK restrictions (e.g. no access to unsafe APIs).

Common situations: Mixed Arrow versions on the classpath from other Spark libraries; running on JDKs where sun.misc.Unsafe access is restricted; shaded/relocated Arrow conflicts.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1496cea2ba9fbad1. Report an issue: GitHub.