We've already touched on Random Forests in this course, but if you're looking for that extra edge in predictive accuracy, Gradient Boosting is where you want to be. While Random Forests build trees in parallel and average them, Gradient Boosting builds them sequentially. Each new tree tries to correct the errors made by the previous ones. It's a powerful approach, but as you'll see, xgboost is a bit more particular about its input than the standard randomForest package.

Let's build a small model to predict loan defaults. I've got some synthetic data here with credit scores, income levels, and loan amounts. Our goal is to predict a binary outcome: 1 for default, 0 for paid back.

Setting up our loan data

# Loading the library
library(xgboost)

# Creating a small synthetic dataset
set.seed(42)
loan_data <- data.frame(
  credit_score = rnorm(100, 650, 50),
  income = rnorm(100, 50000, 10000),
  loan_amount = rnorm(100, 15000, 5000),
  employment_years = sample(1:20, 100, replace = TRUE),
  default = sample(c(0, 1), 100, replace = TRUE)
)

# Splitting into training and testing
train_idx <- sample(1:100, 70)
train_df <- loan_data[train_idx, ]
test_df <- loan_data[-train_idx, ]
---

The "Data Frame" Trap

Here is where I usually trip up when I first switch from caret or randomForest to xgboost. I'll try to pass the data frame directly into the training function because that's what we're used to in R.

# Attempting to train (This will fail)
X_train <- train_df[, -5] # everything but the target
y_train <- train_df$default

model <- xgb.train(
  data = X_train, 
  label = y_train, 
  nrounds = 10, 
  objective = "binary:logistic"
)
---

If you ran that, you'd see an error complaining that the data isn't a matrix or a DMatrix. xgboost is actually a C++ library under the hood, and the R package is a wrapper. It doesn't know how to handle R data.frames directly. It needs the data in a highly optimized format called a DMatrix.

Converting to DMatrix and Training

To fix this, we need to convert our features into a numeric matrix first. I'll use as.matrix() and then wrap it in xgb.DMatrix(). This is the "correct" way to handle the data to ensure the underlying C++ engine can process it efficiently.

# Convert features to a matrix
X_train_mat <- as.matrix(train_df[, -5])
X_test_mat <- as.matrix(test_df[, -5])

# Create the DMatrix objects
dtrain <- xgb.DMatrix(data = X_train_mat, label = train_df$default)
dtest <- xgb.DMatrix(data = X_test_mat, label = test_df$default)

# Now we train. I'll use a low 'eta' (learning rate) to avoid overfitting
# and a 'max_depth' of 3 to keep the trees shallow.
params <- list(
  objective = "binary:logistic",
  eta = 0.1,
  max_depth = 3
)

xgb_model <- xgb.train(
  params = params,
  data = dtrain,
  nrounds = 50,
  watchlist = list(train = dtrain, test = dtest),
  print_every_n = 10,
  early_stopping_rounds = 10
)
---

Notice the watchlist argument. This is one of my favorite features of xgboost. It allows the model to track performance on a validation set in real-time. By pairing it with early_stopping_rounds, we can tell the model: "Stop adding trees if the test error hasn't improved in 10 rounds." This effectively prevents us from overfitting.

Extracting Predictions

Since we used binary:logistic, the model returns probabilities between 0 and 1. To get an actual "Default" or "No Default" classification, we just need to apply a threshold—usually 0.5.

# Predict probabilities
preds_prob <- predict(xgb_model, X_test_mat)

# Convert probabilities to binary classes
preds_class <- ifelse(preds_prob > 0.5, 1, 0)

# Quick check on accuracy
mean(preds_class == test_df$default)
---

It's a straightforward workflow, but the strictness regarding the DMatrix is the most important thing to remember. If you have categorical variables (strings or factors), you'll need to one-hot encode them using something like model.matrix() before converting to a DMatrix, as xgboost only speaks numbers.

📋 Practical Task

Refining a Loan Default Predictor with Hyperparameters

You have been provided with a pre-loaded dataset called loan_data_extended. Your task is to build an xgboost model to predict the default column, but this time you must implement the following specific requirements to improve the model's robustness:

• Convert the features into an xgb.DMatrix.
• Set the learning rate (eta) to 0.05.
• Set the max_depth to 6.
• Train the model for 100 rounds, but implement early stopping that triggers if the validation loss doesn't improve for 15 consecutive rounds.
• Output the final accuracy of the model on the test set.

Ensure you split the data into a 70/30 train/test split before creating your DMatrix objects.