How do I actually turn a block of text into something I can analyze?

The core philosophy of tidytext is that text should be treated like any other data frame. In the standard "tidy" format, each row should be a single token—usually a word. If you have a column of paragraphs, you can't really do any math or grouping on them. You have to "unnest" them.

I've found that unnest_tokens() is the workhorse here. Let's say we have some customer feedback for a local coffee shop:

library(dplyr)
library(tidytext)

feedback <- data.frame(
  id = 1:3,
  text = c(
    "The latte was amazing and the staff were friendly!",
    "The latte was way too hot, but the muffin was delicious.",
    "Slow service today, but the coffee quality is still top-notch."
  )
)

tidy_feedback <- feedback %>%
  unnest_tokens(word, text)

print(tidy_feedback)

Notice what happened: the text column is gone, and we now have a word column. Each row is now a single word. Now we can use all the dplyr tools you already know, like count(), to see which words appear most often.

Why am I seeing words like "the" and "and" dominating my results?

This is the classic text mining headache. These are called "stop words." They are high-frequency words that carry almost no semantic meaning. If you're trying to find out why customers are unhappy, knowing that they used the word "the" 500 times doesn't help you.

The tidytext package comes with a built-in dataset of stop words. The cleanest way to get rid of them is using an anti_join(). It essentially tells R: "Keep every row in my data unless the word exists in the stop_words list."

# Remove common stop words to find the "meat" of the feedback
cleaned_feedback <- tidy_feedback %>%
  anti_join(stop_words)

# Now we can see what actually matters
cleaned_feedback %>%
  count(word, sort = TRUE)

I usually recommend doing this early in your pipeline. There's no point in calculating statistics on words that don't provide any insight.

How do I find words that are unique to a specific review rather than just common overall?

If you're analyzing 1,000 reviews, a word like "coffee" will appear everywhere. It's common, but it's not distinguishing. To find words that characterize a specific document, we use TF-IDF (Term Frequency-Inverse Document Frequency).

Think of TF-IDF as a way to penalize words that appear in every document and reward words that appear frequently in one document but rarely in others. In tidytext, this is handled by bind_tf_idf().

# Let's see which words are most "characteristic" of each review
tfidf_results <- tidy_feedback %>%
  count(id, word, sort = TRUE) %>%
  bind_tf_idf(word, id, n) %>%
  arrange(desc(tf_idf))

head(tfidf_results)

The tf_idf column tells you how important a word is to that specific ID. If "muffin" only appears in review #2, its TF-IDF score will be much higher than "the," which appears in all of them. This is how search engines and recommendation systems often figure out what a page is actually about.

📋 Practical Task

Analyzing Sentiment Contrasts in Restaurant Reviews

You have been given a small dataset of restaurant reviews divided into two categories: "Breakfast" and "Dinner". Your goal is to identify the most characteristic words for each meal time.

reviews <- data.frame(
  meal = c("Breakfast", "Breakfast", "Dinner", "Dinner"),
  text = c(
    "The pancakes were fluffy and the syrup was sweet",
    "Great eggs and coffee, but the toast was burnt",
    "The steak was juicy and the red wine was bold",
    "Amazing pasta and great atmosphere for a date"
  )
)

Write a pipeline that performs the following steps:

1. Unnest the tokens into a column called word.
2. Remove stop words using the stop_words dataset.
3. Calculate the TF-IDF scores, using meal as the document grouping variable.
4. Filter for the top 2 words with the highest tf_idf score for each meal type.

Your final output should be a data frame showing the most distinguishing words for "Breakfast" vs "Dinner".