dotnet/machinelearning · error · InvalidDataException

Cannot read Parquet file

Error message

Cannot read Parquet file

What it means

The Parquet loader opens the stream and calls ParquetReader.Read to build the schema dataset; any exception while parsing the file (corrupt data, unsupported compression/encoding, truncated file) is wrapped in InvalidDataException with message 'Cannot read Parquet file' and the original exception as InnerException.

Solutions

1. Inspect ex.InnerException for the root cause (magic bytes missing, codec not supported, IO error).
2. Verify the file is real Parquet: first 4 bytes must be 'PAR1' (and trailer 'PAR1' at end).
3. Re-download/regenerate the file; ensure the write completed (check file size / checksum).
4. Rewrite the file with a compatible writer/compression (e.g. snappy or uncompressed) using pyarrow/pandas.
5. Upgrade the Parquet.Net dependency to a version supporting the file's features.

Example fix

// before: load directly
var data = mlContext.Data.LoadFromParquetFile(path);
// after: pre-validate then load
if (!File.ReadAllBytes(path)[..4].SequenceEqual("PAR1"u8))
    throw new InvalidDataException($"{path} is not a Parquet file");
var data = mlContext.Data.LoadFromParquetFile(path);

// C#: verify Parquet magic bytes and readability before loading
static void EnsureParquet(string path)
{
    using var fs = File.OpenRead(path);
    Span<byte> head = stackalloc byte[4];
    if (fs.Read(head) < 4 || head.SequenceEqual("PAR1"u8) == false)
        throw new InvalidDataException($"{path} is not a Parquet file");
    fs.Seek(-4, SeekOrigin.End);
    Span<byte> tail = stackalloc byte[4];
    if (fs.Read(tail) < 4 || tail.SequenceEqual("PAR1"u8) == false)
        throw new InvalidDataException($"{path} is truncated");
}

try { var data = mlContext.Data.LoadFromParquetFile(path); }
catch (InvalidDataException ex) when (ex.Message == "Cannot read Parquet file")
{
    Log(ex.InnerException); // codec/version/truncation details
    // re-download or rewrite the file with a compatible writer
}

When it happens

Trigger: Creating a ParquetLoader (via LoadFromParquetFile / parquet data source in Load) on a stream whose bytes are not valid Parquet, a truncated download, an unsupported Parquet feature (newer compression codec, encryption), or an unreadable/closed stream.

Common situations: Pointing the loader at a non-Parquet file (e.g. CSV renamed to .parquet); partially uploaded files; Parquet files written by newer writers using features the bundled Parquet.Net version doesn't support; locked or permission-restricted files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• An 'added_tokens' entry must have a string 'content' and a…
• File at path ' ' cannot be empty
• Replace normalizer has an invalid Regex pattern
• The number of bytes read from stream is less than expected…
• The post-processor ' ' token ' ' with id does not match the…

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/bea29f8a8211eef2. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Parquet/ParquetLoader.cs:167

                    TreatBigIntegersAsDates = args.TreatBigIntegersAsDates
                };

                DataSet schemaDataSet;

                try
                {
                    // We only care about the schema so ignore the rows.
                    ReaderOptions readerOptions = new ReaderOptions()
                    {
                        Count = 0,
                        Offset = 0
                    };
                    schemaDataSet = ParquetReader.Read(stream, _parquetOptions, readerOptions);
                    _rowCount = schemaDataSet.TotalRowCount;
                }
                catch (Exception ex)
                {
                    throw new InvalidDataException("Cannot read Parquet file", ex);
                }

                _columnChunkReadSize = args.ColumnChunkReadSize;
                _columnsLoaded = InitColumns(schemaDataSet);
                Schema = CreateSchema(_host, _columnsLoaded);
            }
        }

        private ParquetLoader(IHost host, ModelLoadContext ctx, IMultiStreamSource files)
        {
            Contracts.AssertValue(host);
            _host = host;
            _host.AssertValue(ctx);
            _host.AssertValue(files);

            // *** Binary format ***
            // int: cached chunk size
            // bool: TreatBigIntegersAsDates flag

View on GitHub (pinned to 7b76e69cf9)