Saturday, October 29, 2011

Plotting gain chart

Gain chart is a popular method to visually inspect model performance in binary prediction. It presents the percentage of captured positive responses as a function of selected percentage of a sample. It is easy to obtain it using ROCR package plotting "tpr" against "rpp". However, it is worth to note that gain chart can be equivalently interpreted as empirical cumulative distribution function of random variable representing rank of randomly selected positive response divided by sample size. This equivalence is presented in the following code:

library(ROCR)
gain.chart <- function(n) {
    score <- runif(n)
    y <- (runif(n) < score)
    plot(performance(prediction(score, y)"tpr""rpp"),
         lwd = 7, main = paste("N =", n))
    lines(ecdf((rank(-score)[y == T]) / n),
          verticals = T, do.points = F, col = "red", lwd = 3)
}

set.seed(1)
par(mfrow = c(12))
gain.chart(10)
gain.chart(10000)

The code plots the following gain charts:


For small samples the two methods do not produce identical plots as ecdf returns step function and ROCR plot provides linear interpolation at jumps.

7 comments:

  1. Have you used Area Under Lift (AUL), analogous to AUC, for model evaluation/comparison? Do you have suggestions on an approach for calculating AUL?

    ReplyDelete
  2. If I understand your question correctly you are asking about the Gini coefficient.

    ReplyDelete
    Replies
    1. I did find this: http://stats.stackexchange.com/questions/24325/lorenz-curve-and-gini-coefficient-for-measuring-classifier-performance

      I will also be looking at calculating the area under the Gain curve as well... (http://www.saedsayad.com/model_evaluation_c.htm)

      Delete
    2. I am looking for methods to calculate the area under the Gain and Lift curves. The flux package has a method using the trapezoid rule (http://artax.karlin.mff.cuni.cz/r-help/library/flux/html/auc.html). What method would you use to add it to your Gain code above?

      Delete
    3. I would use trapezoid rule for approximation of definite integral with data points distributed uniformly (http://en.wikipedia.org/wiki/Trapezoidal_rule).
      Note that there you should set x_1=f(x_1)=0 in order to get the left boundary of the integral.

      Delete
    4. Thanks for the info and reply.

      Delete