{"record":{"id":"e3a3a29cf34472be","repo":"PaddlePaddle/PaddleOCR","slug":"invalid-lmdb-dataset-length-metadata","errorCode":null,"errorMessage":"Invalid LMDB dataset length metadata","messagePattern":"Invalid LMDB dataset length metadata","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"ppocr/data/lmdb_dataset.py","lineNumber":258,"sourceCode":"        return outs\n\n\nclass LMDBDataSetTableMaster(LMDBDataSet):\n    def load_hierarchical_lmdb_dataset(self, data_dir):\n        lmdb_sets = {}\n        dataset_idx = 0\n        env = lmdb.open(\n            data_dir,\n            max_readers=32,\n            readonly=True,\n            lock=False,\n            readahead=False,\n            meminit=False,\n        )\n        txn = env.begin(write=False)\n        num_samples = _restricted_pickle_loads(txn.get(b\"__len__\"))\n        if not isinstance(num_samples, int):\n            raise ValueError(\"Invalid LMDB dataset length metadata\")\n        lmdb_sets[dataset_idx] = {\n            \"dirpath\": data_dir,\n            \"env\": env,\n            \"txn\": txn,\n            \"num_samples\": num_samples,\n        }\n        return lmdb_sets\n\n    def get_img_data(self, value):\n        \"\"\"get_img_data\"\"\"\n        if not value:\n            return None\n        imgdata = np.frombuffer(value, dtype=\"uint8\")\n        if imgdata is None:\n            return None\n        imgori = cv2.imdecode(imgdata, 1)\n        if imgori is None:\n            return None","sourceCodeStart":240,"sourceCodeEnd":276,"githubUrl":"https://github.com/PaddlePaddle/PaddleOCR/blob/2661c7c0ef5c613e8f93c6e93b2e052399f0f854/ppocr/data/lmdb_dataset.py#L240-L276","documentation":"After unpickling the b'__len__' entry, the LMDB loader requires it to be a Python int (the sample count). If the stored value is a string, bytes, or any other type, it raises ValueError('Invalid LMDB dataset length metadata'), refusing to build the dataset cache entry.","triggerScenarios":"Opening an LMDB produced by a different tool where __len__ was stored as e.g. b'10000' (string) or a pickled non-int; or a hand-crafted LMDB missing proper metadata.","commonSituations":"Generating LMDB with a custom/old conversion script that wrote str(n) instead of the pickled int; datasets shared between frameworks (some store text metadata); partial corruption of the metadata record.","solutions":["Verify the type: print(type(_restricted_pickle_loads(txn.get(b'__len__')))) in a scratch script","Regenerate the LMDB with the current PaddleOCR conversion scripts, which pickle the integer count","Patch a custom generator to store the raw int via pickle.dumps(int(n))","Fall back to a label-file based dataset (data_dir + label_file) instead of LMDB if you cannot regenerate"],"exampleFix":"# before (custom generator)\nenv.put(b'__len__', str(n).encode())\n# after\nimport pickle\nenv.put(b'__len__', pickle.dumps(int(n)))","handlingStrategy":"validation","validationCode":"import lmdb, pickle\nn = pickle.loads(lmdb.open(data_dir, readonly=True, lock=False).begin().get(b'__len__'))\nif not isinstance(n, int):\n    raise SystemExit(f'LMDB __len__ has type {type(n).__name__}; regenerate the dataset')","typeGuard":null,"tryCatchPattern":"try:\n    dataset = LMDBDataSet(config, 'Train', logger)\nexcept ValueError as e:\n    if 'Invalid LMDB dataset length' in str(e):\n        raise RuntimeError(f'{data_dir}: __len__ metadata not a pickled int; rebuild LMDB') from e\n    raise","preventionTips":["When writing custom LMDB generators, store pickle.dumps(int(n)) for __len__","Validate a freshly built LMDB by instantiating the dataset class before training","Do not share LMDBs across tools with different metadata conventions"],"tags":["lmdb","dataset","metadata","data-integrity"],"backgroundTag":null,"analyzedSha":"2661c7c0ef5c613e8f93c6e93b2e052399f0f854","analyzedAt":"2026-08-14T20:17:30.180Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}