CountVectorizer vs TfidfVectorizer

Machine learning models such as linear regression, logistic regression, and k-nearest neighbours take in an X and a y variable.

Text data is not already organised as a matrix or vector of real numbers. We say that this data is unstructured.

CountVectorizer converts text into fixed-length vectors by counting how many times each word appears. The tokens are now stored as a bag-of-words.

Limitations:

Tfidf works better than CountVectorizer as it also takes the importance of a words into account account.

Formula → The tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF) which refers to the the number of times a word appears in a document divided by the total number of words in that document. The second term is the Inverse Document Frequency (IDF) which is calculated as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

Limitations:

Related posts:

This is the follow up of this post where I explained what really is the rebase and its golden rule. As many of you pointed it out, most of Anna’s troubles could have been avoided had she used the…

In order to keep ourselves sane we all do some little things in our daily life. Here are some things that work well for me might be helpful for people with the same interest and nature. By going out…

Just a couple days ago, phpMyadmin was once again hit by a new CVE (Common Vulnerability Exposure) this time a SQL injection. For those like me that like to investigate vulnerabilities and ways of…