{"record":{"id":"bea29f8a8211eef2","repo":"dotnet/machinelearning","slug":"cannot-read-parquet-file","errorCode":null,"errorMessage":"Cannot read Parquet file","messagePattern":"Cannot read Parquet file","errorType":"exception","errorClass":"InvalidDataException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Parquet/ParquetLoader.cs","lineNumber":167,"sourceCode":"                    TreatBigIntegersAsDates = args.TreatBigIntegersAsDates\n                };\n\n                DataSet schemaDataSet;\n\n                try\n                {\n                    // We only care about the schema so ignore the rows.\n                    ReaderOptions readerOptions = new ReaderOptions()\n                    {\n                        Count = 0,\n                        Offset = 0\n                    };\n                    schemaDataSet = ParquetReader.Read(stream, _parquetOptions, readerOptions);\n                    _rowCount = schemaDataSet.TotalRowCount;\n                }\n                catch (Exception ex)\n                {\n                    throw new InvalidDataException(\"Cannot read Parquet file\", ex);\n                }\n\n                _columnChunkReadSize = args.ColumnChunkReadSize;\n                _columnsLoaded = InitColumns(schemaDataSet);\n                Schema = CreateSchema(_host, _columnsLoaded);\n            }\n        }\n\n        private ParquetLoader(IHost host, ModelLoadContext ctx, IMultiStreamSource files)\n        {\n            Contracts.AssertValue(host);\n            _host = host;\n            _host.AssertValue(ctx);\n            _host.AssertValue(files);\n\n            // *** Binary format ***\n            // int: cached chunk size\n            // bool: TreatBigIntegersAsDates flag","sourceCodeStart":149,"sourceCodeEnd":185,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Parquet/ParquetLoader.cs#L149-L185","documentation":"The Parquet loader opens the stream and calls ParquetReader.Read to build the schema dataset; any exception while parsing the file (corrupt data, unsupported compression/encoding, truncated file) is wrapped in InvalidDataException with message 'Cannot read Parquet file' and the original exception as InnerException.","triggerScenarios":"Creating a ParquetLoader (via LoadFromParquetFile / parquet data source in Load) on a stream whose bytes are not valid Parquet, a truncated download, an unsupported Parquet feature (newer compression codec, encryption), or an unreadable/closed stream.","commonSituations":"Pointing the loader at a non-Parquet file (e.g. CSV renamed to .parquet); partially uploaded files; Parquet files written by newer writers using features the bundled Parquet.Net version doesn't support; locked or permission-restricted files.","solutions":["Inspect ex.InnerException for the root cause (magic bytes missing, codec not supported, IO error).","Verify the file is real Parquet: first 4 bytes must be 'PAR1' (and trailer 'PAR1' at end).","Re-download/regenerate the file; ensure the write completed (check file size / checksum).","Rewrite the file with a compatible writer/compression (e.g. snappy or uncompressed) using pyarrow/pandas.","Upgrade the Parquet.Net dependency to a version supporting the file's features."],"exampleFix":"// before: load directly\nvar data = mlContext.Data.LoadFromParquetFile(path);\n// after: pre-validate then load\nif (!File.ReadAllBytes(path)[..4].SequenceEqual(\"PAR1\"u8))\n    throw new InvalidDataException($\"{path} is not a Parquet file\");\nvar data = mlContext.Data.LoadFromParquetFile(path);","handlingStrategy":"try-catch","validationCode":"// C#: verify Parquet magic bytes and readability before loading\nstatic void EnsureParquet(string path)\n{\n    using var fs = File.OpenRead(path);\n    Span<byte> head = stackalloc byte[4];\n    if (fs.Read(head) < 4 || head.SequenceEqual(\"PAR1\"u8) == false)\n        throw new InvalidDataException($\"{path} is not a Parquet file\");\n    fs.Seek(-4, SeekOrigin.End);\n    Span<byte> tail = stackalloc byte[4];\n    if (fs.Read(tail) < 4 || tail.SequenceEqual(\"PAR1\"u8) == false)\n        throw new InvalidDataException($\"{path} is truncated\");\n}","typeGuard":null,"tryCatchPattern":"try { var data = mlContext.Data.LoadFromParquetFile(path); }\ncatch (InvalidDataException ex) when (ex.Message == \"Cannot read Parquet file\")\n{\n    Log(ex.InnerException); // codec/version/truncation details\n    // re-download or rewrite the file with a compatible writer\n}","preventionTips":["Validate downloads with checksums before loading.","Keep the Parquet writer library compatible with the reader version (avoid exotic codecs/encryption).","Test load on file creation in pipelines, not at training time.","Handle files atomically: only publish fully-written Parquet files."],"tags":["parquet","file","invalid-data","data-loading"],"backgroundTag":"file-read-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}