{"record":{"id":"c638e9d71826261f","repo":"xtekky/gpt4free","slug":"install-spacy-requirements-pip-install-u-g4f","errorCode":null,"errorMessage":"Install \"spacy\" requirements | pip install -U g4f[files]","messagePattern":"Install \"spacy\" requirements \\| pip install -U g4f\\[files\\]","errorType":"exception","errorClass":"MissingRequirementsError","httpStatus":null,"severity":"warning","filePath":"g4f/tools/files.py","lineNumber":162,"sourceCode":"        if not has_beautifulsoup4:\n            raise MissingRequirementsError(\n                f'Install \"beautifulsoup4\" requirements | pip install -U g4f[files]'\n            )\n        return True\n    elif filename.endswith(\".zip\"):\n        return True\n    elif filename.endswith(\"package-lock.json\") and filename != FILE_LIST:\n        return False\n    else:\n        extension = os.path.splitext(filename)[1][1:]\n        if extension in PLAIN_FILE_EXTENSIONS:\n            return True\n    return False\n\n\ndef spacy_refine_chunks(source_iterator):\n    if not has_spacy:\n        raise MissingRequirementsError(\n            f'Install \"spacy\" requirements | pip install -U g4f[files]'\n        )\n\n    nlp = spacy.load(\"en_core_web_sm\")\n    for page in source_iterator:\n        doc = nlp(page)\n        # for chunk in doc.noun_chunks:\n        #    yield \" \".join([token.lemma_ for token in chunk if not token.is_stop])\n        # for token in doc:\n        #     if not token.is_space:\n        #         yield token.lemma_.lower()\n        #         yield \" \"\n        sentences = list(doc.sents)\n        summary = sorted(sentences, key=lambda x: len(x.text), reverse=True)[:2]\n        for sent in summary:\n            yield sent.text\n\n","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/xtekky/gpt4free/blob/973504e1770928ed5fb82f43da528f441ad9ddc3/g4f/tools/files.py#L144-L180","documentation":"spacy_refine_chunks raises MissingRequirementsError immediately when spacy is not installed. This helper refines document chunks using an NLP model, so it requires the 'spacy' optional dependency plus the 'en_core_web_sm' model (which must be downloaded separately via python -m spacy download).","triggerScenarios":"Calling g4f.tools.files.spacy_refine_chunks (or a code path that uses it for chunk refinement in file/RAG processing) without spacy in the environment. Note: even after installing spacy, spacy.load('en_core_web_sm') will raise OSError if the model wasn't downloaded.","commonSituations":"Using the files extra without the spacy extra; new machine without the downloaded language model; docs/samples that call the chunk refiner without stating its deps.","solutions":["pip install -U 'g4f[files]' then pip install spacy.","Download the required model: python -m spacy download en_core_web_sm.","Verify: python -c \"import spacy; spacy.load('en_core_web_sm')\".","Or skip spacy_refine_chunks and consume the source iterator directly (plain chunking works without spacy)."],"exampleFix":"# before\nfrom g4f.tools.files import spacy_refine_chunks\nchunks = spacy_refine_chunks(pages)  # MissingRequirementsError\n\n# after\n# shell: pip install spacy && python -m spacy download en_core_web_sm\nchunks = spacy_refine_chunks(pages)","handlingStrategy":"validation","validationCode":"def spacy_ready() -> bool:\n    try:\n        import spacy\n        spacy.util.get_installed_models()  # cheap check\n        return 'en_core_web_sm' in spacy.util.get_installed_models()\n    except ImportError:\n        return False\n\nif not spacy_ready():\n    raise SystemExit('Run: pip install spacy && python -m spacy download en_core_web_sm')","typeGuard":"from g4f.errors import MissingRequirementsError\n\ndef can_refine_chunks() -> bool:\n    try:\n        import spacy  # noqa\n        return True\n    except ImportError:\n        return False","tryCatchPattern":"from g4f.errors import MissingRequirementsError\nfrom g4f.tools.files import spacy_refine_chunks\ntry:\n    refined = list(spacy_refine_chunks(pages))\nexcept MissingRequirementsError:\n    refined = pages  # degrade gracefully to raw chunks","preventionTips":["Install spacy AND download en_core_web_sm — the model is a separate step","Guard spacy_refine_chunks with an import check and fall back to raw chunks","Smoke-test model loading at app start","Document the extra deps for any feature using chunk refinement"],"tags":["missing-dependency","spacy","nlp","chunking"],"backgroundTag":null,"analyzedSha":"973504e1770928ed5fb82f43da528f441ad9ddc3","analyzedAt":"2026-08-14T23:45:32.408Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}