{sperrorest} is a generic framework which aims to work with all R models/packages. In statistical learning, model setups, their formulas and error measures all depend on the family of the response variable. Various families exist (numeric, binary, multiclass) which again include sub-families (e.g. gaussian or poisson distribution of a numeric response).
This detail needs to be specified via the respective function,
e.g. when using glm() with a binary response, one needs to
set family = "binomial" to make sure that the model does
something meaningful. Most of the time, the same applies to the generic
predict() function. For the glm() case, one
would need to set type = "response" if the predicted values
should reflect probabilities instead of log-odds.
These settings can be specified using model_args and
pred_args in sperrorest(). So fine, “why do we
need to write all these wrappers and custom model/predict functions
then?!”
model_fun expects at least formula argument and a
data.frame with the learning sample. All arguments, including the
additional ones provided via model_args, are getting passed
to model_fun via a do.call() call. However, if
model_fun does not have an argument named
formula but e.g. fixed (like it is the case
for glmmPQL()) the do.call() call will fail
because sperrorest() tries to pass an argument named
formula but glmmPQL expects an argument named
fixed.
In this case, we need to write a wrapper function for
glmmPQL (named glmmPQL_modelfun here) which
accounts for this naming problem. Here, we are passing the
formula argument to our custom model function which then
does the actual call to glmmPQL() using the supplied
formula object as the fixed argument of
glmmPQL. By default, glmmPQL() has further
arguments like family or random. If we want to
use these, we pass them to model_args which then appends
these to the arguments of glmmPQL_modelfun.
glmmPQL_modelfun <- function(formula = NULL, data = NULL, random = NULL,
family = NULL) {
fit <- glmmPQL(fixed = formula, data = data, random = random, family = family)
return(fit)
}Unless specified explicitly, sperrorest() tries to use
the generic predict() function. This function works
differently depending on the class of the provided fitted model,
i.e. many models slightly differ in the naming (and availability) of
their arguments. For example, when fitting a Support Vector Machine
(SVM) with a binary response variable, package kernlab
expects an argument type = "probabilities" in its
predict() call to receive predicted probabilities while in
package e1071 it is "probability = TRUE".
Similar to model_args, this can be accounted for in the
pred_args of sperrorest().
However, sperrorest() expects that the predicted values
(of any response type) are stored directly in the returned object of the
predict() function. While this is the case for many models,
mainly with a numeric response, classification cases often behave
differently. Here, the predicted values (classes in this case) are often
stored in a sub-object named class or
predicted.
Since there is no way to account for this in a general way (when
every package may return the predicted values in a different
format/column), we need to account for it by providing a custom predict
function which returns only the predicted values so that
sperrorest() can continue properly. This time we are
showing two examples. The first takes again a binary classification
using randomForest.
When calling predict on a fitted randomForest model with
a binary response variable, the predicted values are actually stored in
the resulting object returned by predict() (here called
pred). So why do we have trouble here then?
Simply because pred is a matrix containing both
probabilities for the FALSE (= 0) and TRUE (=
1) case. sperrorest() needs a vector containing only the
predicted values of the TRUE case to pass these further
onto err_fun() which then takes care of calculating all the
error measures. So the important part is to subset the resulting matrix
in the pred object to TRUE cases only and
return the result.
rf_predfun <- function(object = NULL, newdata = NULL, type = NULL) {
pred <- predict(object = object, newdata = newdata, type = type)
pred <- pred[, 2]
}The same case (binary response) using svm from the
e1071 package. Here, the predicted probabilities are stored
in a sub-object of pred. We can address it using the
attr() function. Then again, we only need the
TRUE cases for sperrorest().
svm_predfun <- function(object = NULL, newdata = NULL, probability = NULL) {
pred <- predict(object, newdata = newdata, probability = TRUE)
pred <- attr(pred, "probabilities")[, 2]
}