You've probably spent the last few lessons cleaning your text data—removing stop words, stemming, and getting everything into a nice, tidy tibble. Now you want to find the "hidden" themes in your documents using Latent Dirichlet Allocation (LDA). But when you try to run your first model, you're likely to hit a wall like this:

library(topicmodels)
library(tidytext)

# Assume 'reviews_tidy' is a tidy tibble with 'document' and 'term' columns
lda_model <- LDA(reviews_tidy, k = 5, control = list(seed = 123))

# Error in .local(x, k, ...) : x must be a DocumentTermMatrix

The Matrix Mismatch

This is the most common stumbling block I see. You're using the tidytext ecosystem, which loves "long" data (one word per row). However, the topicmodels package is older and expects a DocumentTermMatrix (DTM). A DTM is essentially a giant spreadsheet where every row is a document and every column is a unique word in your entire vocabulary. The LDA() function cannot read a tibble; it needs that specific matrix object.

To fix this, we use cast_dtm(). This function takes your tidy data and "pivots" it into the format the LDA algorithm requires. Here is the corrected flow:

# Convert the tidy tibble into a DocumentTermMatrix
dtm <- reviews_tidy %>%
  cast_dtm(term = term, document = document)

# Now pass the DTM into the LDA function
lda_model <- LDA(dtm, k = 5, control = list(seed = 123))

By casting the data first, you've transformed your list of words into a mathematical representation that the model can actually compute. I usually recommend checking the dimensions of your DTM with dim(dtm) before running the model—if you have 50,000 unique words, your model might crawl to a halt.

Interpreting the Probabilities

Once the model runs, you'll realize LDA doesn't just "tell" you what the topics are. It doesn't name them "Customer Service" or "Pricing." Instead, it gives you two sets of probabilities: $\beta$ (beta) and $\gamma$ (gamma). I like to think of these as the "What" and the "Where."

• $\beta$ (Topic-Word probabilities): This tells you which words are most associated with which topic. For example, if Topic 1 has high probabilities for "battery," "charge," and "power," you can manually label that topic as "Battery Life."
• $\gamma$ (Document-Topic probabilities): This tells you how much of each document is composed of each topic. A single review might be 70% "Battery Life" and 30% "Shipping Speed."

To get these into a usable format, use the tidy() function from tidytext. It turns those complex model objects back into the tidy tibbles we actually enjoy working with.

# Get the word-topic probabilities (beta)
topics <- tidy(lda_model, matrix = "beta")

# Get the document-topic probabilities (gamma)
doc_topics <- tidy(lda_model, matrix = "gamma")

The Art of Choosing k

The biggest headache in LDA is picking k, the number of topics. If k is too small, your topics will be overly broad (e.g., everything is just "General Feedback"). If k is too large, you'll get redundant topics that are nearly identical.

There isn't a magic button for this, but I usually follow a "bracket" approach. I'll run the model with 3, 5, 10, and 20 topics. I then look at the top 10 words for each topic. When the topics start feeling too fragmented or the same words keep appearing in every single topic, I know I've pushed k too far. It's an iterative process—more of an art than a hard science.

📋 Practical Task

Uncovering Themes in App Store Reviews

You have been given a dataset of 500 user reviews for a fictional fitness app. Your goal is to identify the top 3 themes users are discussing without reading every single review.

Your Task:

1. Create a small sample tidy tibble (or use a provided one) with columns document and term.
2. Convert this tidy data into a DocumentTermMatrix using cast_dtm().
3. Fit an LDA model with k = 3.
4. Use the tidy() function to extract the $\beta$ matrix.
5. For each of the 3 topics, identify the top 5 words with the highest probability.
6. Based on those words, assign a human-readable label to each topic (e.g., "Bug Reports," "Feature Requests," "User Praise").