{"record":{"id":"8767b8bd4affb44f","repo":"jd-opensource/joyagent-jdgenie","slug":"the-dataframe-data-is-empty-after-dropna","errorCode":null,"errorMessage":"The dataframe {data} is empty after dropna.","messagePattern":"The dataframe (.+?) is empty after dropna\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"genie-tool/genie_tool/tool/analysis_component/data_model.py","lineNumber":78,"sourceCode":"class DataModel(BaseModel):\n    id: str = Field(str(uuid.uuid4()), description=\"\")\n    measure: Measure = Field(None, description=\"度量\")\n    data: pd.DataFrame = Field(exclude=True)\n    columns: List[Column] = Field(\n        [], description=\"可分析的维度\", validate_default=True)\n\n    model_config = ConfigDict(arbitrary_types_allowed=True)\n\n    def __len__(self):\n        return len(self.data)\n\n    @field_validator(\"data\", mode=\"before\")\n    @classmethod\n    def validate_data(cls, data: pd.DataFrame, values) -> pd.DataFrame:\n        measure = values.data[\"measure\"]\n        data = data.dropna(subset=[measure.column])\n        if len(data) == 0:\n            raise ValueError(f\"The dataframe {data} is empty after dropna.\")\n        for col in data.columns:\n            if len(data[col].unique()) == 1:\n                data = data.drop(col, axis=1)\n        return data\n\n    @field_validator(\"columns\", mode=\"before\")\n    @classmethod\n    def validate_columns(cls, val, values) -> List[Column]:\n        data = values.data[\"data\"]\n        measure = values.data[\"measure\"]\n        if not val:\n            val = [Column(name=c,\n                          is_series=is_datetime64_any_dtype(data[c]),\n                          is_number=is_numeric_dtype(data[c]),\n                          ) for c in data.columns if c != measure.column]\n        if val and isinstance(val, list) and isinstance(val[0], str):\n            val = [c for c in val if c in data.columns] or data.columns\n            val = [Column(name=c,","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/jd-opensource/joyagent-jdgenie/blob/2417e0b8b636d941ad5fb14c59b20dddfef5375d/genie-tool/genie_tool/tool/analysis_component/data_model.py#L60-L96","documentation":"A pydantic `field_validator` on the `data` DataFrame drops rows with NaN in the measure column, and raises this ValueError if nothing remains. It is raised during model construction, so instantiating/validating the data model fails whenever the measure column is entirely missing or null in the relevant subset.","triggerScenarios":"Constructing the analysis data model with a DataFrame where the configured `measure.column` has all-NaN values, or the column name is misspelled so dropna removes every row.","commonSituations":"Upstream query returned no/blank data for the measure; measure column name mismatch between config and dataframe; filtered rows removed all observations; schema drift after source change.","solutions":["Check the measure column name in your config matches the DataFrame column exactly","Inspect the DataFrame before model construction: confirm the measure column has non-null values","Fix the upstream query/filter so it returns rows, or handle empty input before building the model","Drop or repair rows with NaN in the measure column at ingestion"],"exampleFix":"// before\ndf = fetch_data(); model = AnalysisDataModel(data=df, columns=cols)  # measure col all NaN\n// after\ndf = df.dropna(subset=[measure_col])\nif df.empty:\n    raise ValueError(f\"No data for measure column {measure_col}\")\nmodel = AnalysisDataModel(data=df, columns=cols)","handlingStrategy":"validation","validationCode":"if df.empty or df[measure_col].isna().all():\n    raise ValueError(f\"No usable data in measure column '{measure_col}'\")\nif measure_col not in df.columns:\n    raise ValueError(f\"Measure column '{measure_col}' missing from dataframe\")","typeGuard":"def has_measure_data(df, measure_col: str) -> bool:\n    return measure_col in df.columns and df[measure_col].notna().any()","tryCatchPattern":"try:\n    model = AnalysisDataModel(data=df, columns=cols)\nexcept ValueError as e:\n    if \"empty after dropna\" in str(e):\n        fix_upstream_query_or_alert()\n    raise","preventionTips":["Verify the measure column name matches between config and source schema","Check for NaNs in the measure column right after data load","Alert on empty upstream query results before model construction"],"tags":["pandas","pydantic","data-validation"],"backgroundTag":"empty-result-set","analyzedSha":"2417e0b8b636d941ad5fb14c59b20dddfef5375d","analyzedAt":"2026-09-08T11:28:19.414Z","contentChangedAt":"2026-09-08T11:28:19.414Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}