dotnet/machinelearning · error · NotImplementedException

{fieldType.Name}

Error message

{fieldType.Name}

What it means

FromArrowRecordBatch / AppendDataFrameColumnFromArrowArray throw NotImplementedException when an Arrow record batch contains a field whose ArrowTypeId has no mapping to a DataFrameColumn. Unsupported type ids include Map, Null, Time32, Time64 and others that hit the switch's default branch.

Solutions

1. Preprocess the Arrow data upstream so unsupported fields (Map, Null, Time32, Time64) are converted to supported types (e.g. cast Time32/Time64 to Timestamp or Int64, Map to List/Struct).
2. Filter unsupported columns out of the Arrow schema before calling FromArrowRecordBatch.
3. Convert the batch to IPC/records yourself and build columns manually with DataFrameColumn.Create for unsupported types.
4. Check the Apache.Arrow version to see if newer mappings were added, and upgrade the package.

Example fix

// before
var df = DataFrame.FromArrowRecordBatch(batch); // batch has Time32 column -> NotImplementedException
// after
var supportedSchema = batch.Schema.RemoveField(batch.Schema.GetFieldIndex("myTime"));
var df = DataFrame.FromArrowRecordBatch(new Apache.Arrow.RecordBatch(supportedSchema, /* supported arrays only */));

var unsupported = batch.Schema.Fields
    .Where(f => f.DataType.Type is ArrowTypeId.Map or ArrowTypeId.Null or ArrowTypeId.Time32 or ArrowTypeId.Time64)
    .Select(f => f.Name)
    .ToList();
if (unsupported.Count > 0) throw new InvalidOperationException($"Unsupported Arrow columns: {string.Join(',', unsupported)}");

static bool IsArrowTypeSupported(ArrowTypeId t) =>
    t is not (ArrowTypeId.Map or ArrowTypeId.Null or ArrowTypeId.Time32 or ArrowTypeId.Time64);

try
{
    var df = DataFrame.FromArrowRecordBatch(batch);
}
catch (NotImplementedException ex)
{
    // ex.Message contains the Arrow field type name; convert/filter that column and retry
}

When it happens

Trigger: Calling DataFrame.FromArrowRecordBatch on a record batch whose schema contains Arrow columns of type Map, Null, Time32, Time64, or any other type not handled by the switch in AppendDataFrameColumnFromArrowArray.

Common situations: Reading Arrow data produced by systems that emit time32/time64 or map columns (Spark, Polars, Pandas exports); schema changes upstream introducing new field types; round-tripping data types that DataFrame does not model.

Related errors

• nameof(T)
• type.ToString()
• NotImplementedException
• offsetBuffer
• Strings.InconsistentNullBitMapAndLength

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/277e0169a13ddb2b. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.Data.Analysis/DataFrame.Arrow.cs:141

                        dataFrameColumn = dataTimeDataFrameColumn;
                    }
                    break;
                case ArrowTypeId.Decimal128:
                case ArrowTypeId.Decimal256:
                case ArrowTypeId.Binary:
                case ArrowTypeId.Date32:
                case ArrowTypeId.Dictionary:
                case ArrowTypeId.FixedSizedBinary:
                case ArrowTypeId.HalfFloat:
                case ArrowTypeId.Interval:
                case ArrowTypeId.List:
                case ArrowTypeId.Map:
                case ArrowTypeId.Null:
                case ArrowTypeId.Time32:
                case ArrowTypeId.Time64:

                default:
                    throw new NotImplementedException($"{fieldType.Name}");
            }

            if (dataFrameColumn != null)
            {
                ret.Columns.Insert(ret.Columns.Count, dataFrameColumn);
            }
        }

        /// <summary>
        /// Wraps a <see cref="DataFrame"/> around an Arrow <see cref="RecordBatch"/> without copying data
        /// </summary>
        /// <param name="recordBatch"></param>
        /// <returns><see cref="DataFrame"/></returns>
        public static DataFrame FromArrowRecordBatch(RecordBatch recordBatch)
        {
            DataFrame ret = new DataFrame();
            Apache.Arrow.Schema arrowSchema = recordBatch.Schema;
            int fieldIndex = 0;

View on GitHub (pinned to 7b76e69cf9)