Saturday, October 29, 2011

Plotting gain chart

Gain chart is a popular method to visually inspect model performance in binary prediction. It presents the percentage of captured positive responses as a function of selected percentage of a sample. It is easy to obtain it using ROCR package plotting "tpr" against "rpp". However, it is worth to note that gain chart can be equivalently interpreted as empirical cumulative distribution function of random variable representing rank of randomly selected positive response divided by sample size. This equivalence is presented in the following code:

library(ROCR)
gain.chart <- function(n) {
    score <- runif(n)
    y <- (runif(n) < score)
    plot(performance(prediction(score, y)"tpr""rpp"),
         lwd = 7, main = paste("N =", n))
    lines(ecdf((rank(-score)[y == T]) / n),
          verticals = T, do.points = F, col = "red", lwd = 3)
}

set.seed(1)
par(mfrow = c(12))
gain.chart(10)
gain.chart(10000)

The code plots the following gain charts:


For small samples the two methods do not produce identical plots as ecdf returns step function and ROCR plot provides linear interpolation at jumps.

6 comments:

  1. Have you used Area Under Lift (AUL), analogous to AUC, for model evaluation/comparison? Do you have suggestions on an approach for calculating AUL?

    ReplyDelete
  2. If I understand your question correctly you are asking about the Gini coefficient.

    ReplyDelete
    Replies
    1. I would use trapezoid rule for approximation of definite integral with data points distributed uniformly (http://en.wikipedia.org/wiki/Trapezoidal_rule).
      Note that there you should set x_1=f(x_1)=0 in order to get the left boundary of the integral.

      Delete
    2. Thanks for the info and reply.

      Delete
    3. Use ROCR package in R
      use function performance

      example

      library(ROCR)
      pred<-prediction(actual,predicted)
      perf<-performance(pred,"tpr","fpr")
      plot(perf,col="red")
      abline(0,1, lty = 8, col = "grey")

      auc<-performance(pred,"auc")
      unlist(auc@y.values)

      Delete

Note: Only a member of this blog may post a comment.