An efficient framework for online advertising effectiveness measurement and comparison

ABSTRACT

In online advertising market it is crucial to provide advertisers with a reliable measurement of advertising effectiveness to make better marketing campaign planning. The basic idea for ad effectiveness measurement is to compare the performance (e.g., success rate) among users who were and who were not exposed to a certain treatment of ads. When a randomized experiment is not available, a naive comparison can be biased because exposed and unexposed populations typically have different features. One solid methodology for a fair comparison is to apply inverse propensity weighting with doubly robust estimation to the observational data. However the existing methods were not designed for the online advertising campaign, which usually suffers from huge volume of users, high dimensionality, high sparsity and imbalance. We propose an efficient framework to address these challenges in a real campaign circumstance. We utilize gradient boosting stumps for feature selection and gradient boosting trees for model fitting, and propose a subsampling-and-backscaling procedure that enables analysis on extremely sparse conversion data. The choice of features, models and feature selection scheme are validated with irrelevant conversion test. We further propose a parallel computing strategy, combined with the subsampling-and-backscaling procedure to reach computational efficiency. Our framework is applied to an online campaign involving millions of unique users, which shows substantially better model fitting and efficiency. Our framework can be further generalized to comparison of multiple treatments and more general treatment regimes, as sketched in the paper. Our framework is not limited to online advertising, but also applicable to other circumstances (e.g., social science) where a 'fair' comparison is needed with observational data.

References

1. Apache™ hadoop® project. http://hadoop.apache.org.
2. H. Bang and J. M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962--973, 2005.
3. J. Barajas, J. Kwon, R. Akella, A. Flores, M. Holtan, and V. Andrei. Marketing campaign evaluation in targeted display advertising. In Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy, page 5. ACM, 2012.
4. A. Basu, D. Polsky, and W. G. Manning. Use of propensity scores in non-linear response models: the case for health care expenditures. Technical report, National Bureau of Economic Research, 2008.
5. D. Chan, R. Ge, O. Gershony, T. Hesterberg, and D. Lambert. Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 7--16. ACM, 2010.
6. D. R. Cox. Planning of experiments. 1958.
7. B. Dalessandro, C. Perlich, O. Stitelman, and F. Provost. Causally motivated attribution for online advertising. In Proceedings of the Sixth International Workshop on Data Mining for Online Advertising and Internet Economy, page 7. ACM, 2012.
8. A. Dasgupta, K. Punera, J. M. Rao, X. Wang, J. Rao, and X.-J. Wang. Impact of spam exposure on user engagement. In USENIX Security, 2012.
9. J. H. Friedman. Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4):367--378, 2002.
10. M. J. Funk, D. Westreich, C. Wiesen, T. Stürmer, M. A. Brookhart, and M. Davidian. Doubly robust estimation of causal effects. American journal of epidemiology, 173(7):761--767, 2011.
11. Greg Ridgeway with contributions from others. gbm: Generalized Boosted Regression Models, 2013. R package version 2.0--8.
12. S. Guo and M. W. Fraser. Propensity score analysis. Statistical methods and applications, 2010.
13. J. J. Heckman, H. Ichimura, and P. Todd. Matching as an econometric evaluation estimator. The Review of Economic Studies, 65(2):261--294, 1998.
14. R. C. Holte. Very simple classification rules perform well on most commonly used datasets. Machine learning, 11(1):63--90, 1993.
15. K. Imai. Do get-out-the-vote calls reduce turnout? the importance of statistical methods for field experiments. American Political Science Review, 99(2):283--300, 2005.
16. K. Imai and D. A. Van Dyk. Causal inference with general treatment regimes. Journal of the American Statistical Association, 99(467), 2004.
17. L. Kish. Survey sampling. new york: J. Wiley & Sons, 643:16, 1965.
18. M. Lechner. Earnings and employment effects of continuous gff-the-job training in east germany after unification. Journal of Business & Economic Statistics, 17(1):74--90, 1999.
19. J. Ledolter. Multinomial logistic regression. Data Mining and Business Analytics with R, pages 132--149.
20. S. F. Lehrer and G. Kordas. Matching using semiparametric propensity scores. Empirical Economics, 44(1):13--45, 2013.
21. R. A. Lewis, J. M. Rao, and D. H. Reiley. Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising. In Proceedings of the 20th international conference on World wide web, pages 157--166. ACM, 2011.
22. D. F. McCaffrey, G. Ridgeway, A. R. Morral, et al. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological methods, 9(4):403--425, 2004.
23. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2008. ISBN 3-900051-07-0.
24. G. Ridgeway. Generalized boosted models: A guide to the gbm package. Update, 1:1.
25. J. M. Robins, A. Rotnitzky, and L. P. Zhao. Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association, 90(429):106--121, 1995.
26. P. R. Rosenbaum. Observational studies. Springer, 2002.
27. P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41--55, 1983.
28. P. R. Rosenbaum and D. B. Rubin. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387):516--524, 1984.
29. P. R. Rosenbaum and D. B. Rubin. Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1):33--38, 1985.
30. D. O. Scharfstein, A. Rotnitzky, and J. M. Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94(448):1096--1120, 1999.
31. O. Stitelman, B. Dalessandro, C. Perlich, and F. Provost. Estimating the effect of online display advertising on browser conversion. Data Mining and Audience Intelligence for Advertising (ADKDD 2011), 8, 2011.
32. P. Wang, M. Traskin, and D. S. Small. Robust inferences from a before-and-after study with multiple unaffected control groups. Journal of Causal Inference, pages 1--26, 2013.