How do I actually turn a bunch of raw text into a sentiment score?

The most common way to handle this in R is through a "lexicon" approach. Think of a lexicon as a giant dictionary where every word is pre-tagged as positive or negative. Instead of training a complex machine learning model, we just break our text into individual words and count how many of them match the dictionary.

I usually lean on the tidytext package because it plays nicely with the tidyverse. Let's say we're analyzing customer reviews for a fictional coffee shop, "The Roasted Bean."

library(dplyr)
library(tidytext)

# A few sample reviews
reviews <- data.frame(
  id = 1:3,
  text = c("The latte was amazing and the staff were friendly!", 
           "Slow service and the muffin was dry. Disappointing.", 
           "Average coffee, but the atmosphere is cozy.")
)

# Tokenize the text (one word per row)
review_words <- reviews %>%
  unnest_tokens(word, text)

# Join with the 'bing' lexicon (positive/negative)
sentiment_counts <- review_words %>%
  inner_join(get_sentiments("bing")) %>%
  count(id, sentiment)

print(sentiment_counts)

In this workflow, we've stripped away the "noise" and kept only the words that carry emotional weight. It's simple, but it's surprisingly effective for high-volume data.

Which lexicon should I use? Bing, AFINN, or NRC?

You'll see these three pop up constantly. The "right" one depends entirely on what you're trying to prove. I've wasted a few afternoons using the wrong one, so here is the cheat sheet.

• Bing: This is binary. It's either "positive" or "negative." Use this if you just need a general vibe check or a simple pie chart.
• AFINN: This assigns a score from -5 to +5. If you need to know how angry or how happy a customer is, this is your go-to. It gives you a weighted average rather than just a count.
• NRC: This is the most complex. It categorizes words into emotions like "trust," "fear," "joy," and "sadness." It's great for deeper psychological analysis, but it's overkill for a basic sentiment check.

If you're just starting out, stick with Bing. It's the easiest to debug and visualize.

What about words like "not good"? Does R just see "good" and call it positive?

You've hit on the biggest weakness of the "bag-of-words" approach. If a customer writes "the coffee was not good," a standard inner_join with a lexicon sees the word "good" and marks it as positive. It completely ignores the "not."

To fix this, you need to handle "valence shifters." This is where I stop using tidytext and switch to the sentimentr package. It looks at the words surrounding the sentiment word to see if the meaning has been flipped.

library(sentimentr)

text_data <- c("The coffee was good", "The coffee was not good")
sentiment_scores <- sentiment(text_data)

# The first will be positive, the second will be negative.
print(sentiment_scores$sentiment)

sentimentr is much more computationally expensive than a simple join, but if your dataset is full of nuances, sarcasm, or negations, it's the only way to get an honest result.

📋 Practical Task

Analyzing Hotel Guest Feedback for 'The Grand Vista'

You have been handed a small dataset of guest reviews for "The Grand Vista" hotel. Your goal is to determine the overall sentiment of the guest experience using the tidytext and dplyr packages.

Your Task:

1. Create a data frame containing these three reviews:
   • "The room was spacious and the view was breathtaking!"
   • "I hated the noisy hallways and the breakfast was cold."
   • "The staff were helpful, but the check-in process was a nightmare."
2. Tokenize the reviews using unnest_tokens().
3. Join the tokens with the "bing" sentiment lexicon.
4. Calculate a "Net Sentiment Score" for the hotel by subtracting the total number of negative words from the total number of positive words across all reviews.

Hint: Your final result should be a single numeric value.