Which command is required to obtain the language-specific data files necessary for spaCy to process English text after the library itself is installed?

While 'pip install' installs the library code, spaCy requires a separate command to download the trained models and language data (like 'en_core_web_sm') used for tokenization and other NLP tasks.

What is the primary characteristic of stemming that distinguishes it from lemmatization?

Stemming is a rule-based approach that chops off suffixes (like -ing or -ed) using fixed heuristics. Unlike lemmatization, it does not use a dictionary or linguistic knowledge, which is why it can result in non-words like 'abil' from 'ability'.

What is the primary function of tokenization in a Natural Language Processing pipeline?

Tokenization is the foundational step that splits text into individual tokens (words, punctuation, etc.) so that the machine can process them. Translating is a different task, reducing words to roots is lemmatization, and assigning categories is part-of-speech tagging.

If a specific term appears in every single document in a large corpus, what is the likely impact on its TF-IDF score?

If a word is present in every document, the ratio of 'total documents' to 'documents containing the term' becomes 1. The log of 1 is 0, which results in a very low IDF score, effectively ignoring the word during classification.

Learn Natural Language Processing

Learn Natural Language Processing - Free Learning Path