Importance Score Computation for Elastic Net

April 10, 1019

Caution to use variable importance score with caret.

According to the caret package, it is supposed to provide absolute value of the regression coefficients (excluding intercept).

“glmboost and glmnet: the absolute value of the coefficients corresponding the the tuned model are used.”
Below is the actual function that is supposed to be used to compute varImp

caret::getModelInfo("glmnet")$glmnet$varImp

## function(object, lambda = NULL, ...) {
##                     if(is.null(lambda)) {
##                       if(length(lambda) > 1) stop("Only one value of lambda is allowed right now")
##                       if(!is.null(object$lambdaOpt)) {
##                         lambda <- object$lambdaOpt
##                       } else stop("must supply a value of lambda")
##                     }
##                     beta <- predict(object, s = lambda, type = "coef")
##                     if(is.list(beta)) {
##                       out <- do.call("cbind", lapply(beta, function(x) x[,1]))
##                       out <- as.data.frame(out)
##                     } else out <- data.frame(Overall = beta[,1])
##                     out <- abs(out[rownames(out) != "(Intercept)",,drop = FALSE])
##                     out
##                   }

However, the following example showed the wierd pattern: the largest value of the variable importance of the ElasticNet is always 100!

set.seed(998)
data(Sonar)   #Random data, just for illustration purpose
inTraining <- createDataPartition(Sonar$Class, p = 0.75, list = FALSE)
training <- Sonar[inTraining, ]
testing <- Sonar[-inTraining, ]
modelFit <- train( V6~.,data=training, method="glmnet" ,trControl =trainControl(method = "cv", number = 5))
varImp(modelFit)

## glmnet variable importance
## 
##   only 20 most important variables shown (out of 60)
## 
##       Overall
## V7  1.000e+02
## V5  8.161e+01
## V52 5.367e+01
## V50 2.572e+01
## V13 7.533e+00
## V32 1.508e+00
## V12 1.164e+00
## V38 9.790e-01
## V47 2.576e-01
## V31 1.837e-03
## V53 0.000e+00
## V35 0.000e+00
## V57 0.000e+00
## V9  0.000e+00
## V16 0.000e+00
## V21 0.000e+00
## V2  0.000e+00
## V49 0.000e+00
## V51 0.000e+00
## V48 0.000e+00

We investigated what’s happening here. We found that varImp function automatically scale the regression coefficients with the largest coefficient. Use the varImp with caution!

my.glmnet.model <- modelFit$finalModel
coefs=abs(coef(my.glmnet.model, s = modelFit$bestTune$lambda))

indx=sort.int(abs(as.vector(coefs))[-1],index.return=TRUE,decreasing=TRUE)$ix

data.frame(importance=round(coefs[indx+1,]/max(coefs)*100,2)) %>% 
  tibble::rownames_to_column( var = "rowname") %>% dplyr::filter(importance>0)

##   rowname importance
## 1      V7     100.00
## 2      V5      81.61
## 3     V52      53.67
## 4     V50      25.72
## 5     V13       7.53
## 6     V32       1.51
## 7     V12       1.16
## 8     V38       0.98
## 9     V47       0.26