Practical collapsed variational bayes inference for hierarchical dirichlet process

ABSTRACT

We propose a novel collapsed variational Bayes (CVB) inference for the hierarchical Dirichlet process (HDP). While the existing CVB inference for the HDP variant of latent Dirichlet allocation (LDA) is more complicated and harder to implement than that for LDA, the proposed algorithm is simple to implement, does not require variance counts to be maintained, does not need to set hyper-parameters, and has good predictive performance.

References

1. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993--1022, 2003.
2. M. Rosen-Zvi, T. Griffiths, M. Steyvers, and P. Smyth. The author-topic model for authors and documents. In Proceedings of the 20th conference on Uncertainty in artificial intelligence, pages 487--494. AUAI Press, 2004.
3. M. Steyvers, P. Smyth, M. Rosen-Zvi, and T. Griffiths. Probabilistic author-topic models for information discovery. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 306--315. ACM Press, 2004.
4. D. Newman, C. Chemudugunta, and P. Smyth. Statistical entity-topic models. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 680--686. ACM Press, 2006.
5. R. M. Nallapati, A. Ahmed, E. P. Xing, and W. W. Cohen. Joint latent topic models for text and citations. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 542--550. ACM, 2008.
6. A. Gruber, M. Rosen-Zvi, and Y. Weiss. Latent Topic Models for Hypertext. In UAI, pages 230--239, 2008.
7. A. Ahmed, E. P. Xing, W. W. Cohen, and R. F. Murphy. Structured correspondence topic models for mining captioned figures in biological literature. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '09, pages 39--48. ACM, 2009.
8. R. Collobert, F. Sinz, J. Weston, and L. Bottou. Trading convexity for scalability. In Proceedings of the 23rd international conference on Machine learning, pages 201--208. ACM, 2006.
9. D. M. Blei and J. D. Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113--120. ACM, 2006.
10. L. Yao, D. Mimno, and A. McCallum. Efficient methods for topic model inference on streaming document collections. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '09, pages 937--946. ACM, 2009.
11. T. Iwata, T. Yamada, Y. Sakurai, and N. Ueda. Online multiscale dynamic topic models. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 663--672. ACM, 2010.
12. L. Hong, D. Yin, J. Guo, and B. D. Davison. Tracking trends: incorporating term volume into temporal topic models. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '11, pages 484--492. ACM, 2011.
13. I. Sato and H. Nakagawa. Topic Models with Power-Law Using Pitman-Yor Process. In Proceedings of 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2010.
14. D. Ramage, C. D. Manning, and S. Dumais. Partially labeled topic models for interpretable text mining. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '11, pages 457--465. ACM, 2011.
15. J. Zhu, N. Lao, N. Chen, and E. P. Xing. Conditional topical coding: an efficient topic model conditioned on rich features. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '11, pages 475-483. ACM, 2011.
16. X. Wei and W. B. Croft. LDA-based document models for ad-hoc retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR '06, pages 178--185. ACM, 2006.
17. D. Andrzejewski and D. Buttler. Latent topic feedback for information retrieval. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '11, pages 600--608. ACM, 2011.
18. K. El-Arini, G. Veda, D. Shahaf, and C. Guestrin. Turning down the noise in the blogosphere. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '09, pages 289--298. ACM, 2009.
19. J. Chang, J. L. Boyd-Graber, and D. M. Blei. Connections between the lines: augmenting social networks with text. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, June 28-July 1, 2009, pages 169--178. ACM, 2009.
20. S. Gerrish and D. M. Blei. A Language-based Approach to Measuring Scholarly Impact. In Proceedings of the 27th International Conference on Machine Learning (ICML2010), year 2010, pages 375--382.
21. S. Gerrish and D. M. Blei. Predicting Legislative Roll Calls from Text. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pages 489--496, 2011.
22. C. Wang and D. M. Blei. Collaborative topic modeling for recommending scientific articles. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 448--456, 2011.
23. Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei. Hierarchical Dirichlet Processes. Journal of the American Statistical Association, 101(476):1566--1581, 2006.
24. Y. W. Teh, K. Kurihara, and M. Welling. Collapsed Variational Inference for HDP. In Advances in Neural Information Processing Systems 20, 2008.
25. Y. W. Teh, D. Newman, and M. Welling. A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems 19, 2007.
26. A. Asuncion, M. Welling, P. Smyth, and Y. W. Teh. On Smoothing and Inference for Topic Models. In Proceedings of the International Conference on Uncertainty in Artificial Intelligence, 2009.
27. A. Asuncion. Approximate Mean Field for Dirichlet-Based Models. In Topic Models Workshop, ICML.
28. H. Wallach, D. Mimno, and A. McCallum. Rethinking LDA: Why Priors Matter. In Advances in Neural Information Processing Systems 22, pages 1973--1981. 2009.
29. H. Ishwaran and L. F. James. Gibbs Sampling Methods for Stick Breaking Priors. Journal of the American Statistical Association, 96(453):161--173, 2001.
30. Escobar and West. Bayesian Density Estimation and Inference using Mixtures. Journal of the American Statistical Association, 90, 1995.
31. D. Sontag and D. Roy. Complexity of Inference in Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems 24, pages 1008--1016. 2011.
32. T. P. Minka. Estimating a Dirichlet distribution. Technical report, Microsoft, 2000.
33. C. Elkan. Clustering documents with an exponential-family approximation of the Dirichlet compound multinomial distribution. In Proceedings of the 23rd international conference on Machine learning, pages 289--296, 2006.
34. A. Frank and A. Asuncion. UCI Machine Learning Repository, 2010.
35. H. Deng, J. Han, B. Zhao, Y. Yu, and C. X. Lin. Probabilistic topic models with biased propagation on heterogeneous information networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1271--1279, 2011.
36. J. Leskovec, J. Kleinberg, and C. Faloutsos. Graphs over time: densification laws, shrinking diameters and possible explanations. In Proceedings of the 11th ACM SIGKDD international conference on Knowledge discovery in data mining, KDD '05, pages 177--187. ACM, 2005.
37. J. Leskovec, J. Kleinberg, and C. Faloutsos. Graph evolution: Densification and shrinking diameters. ACM Transactions on Knowledge Discovery from Data (ACM TKDD), 1(1), 2007.
38. M. D. Hoffman, D. M. Blei, and F. R. Bach. Online Learning for Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, pages 856--864, 2010.
39. I. Sato, K. Kurihara, and H. Nakagawa. Deterministic Single-Pass Algorithm for LDA. In J. Lafferty, C. K. I. Williams, R. Zemel, J. Shawe-Taylor, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 2074--2082. 2010.
40. C. Wang, J. W. Paisley, and D. M. Blei. Online Variational Inference for the Hierarchical Dirichlet Process. Journal of Machine Learning Research - Proceedings Track, 15:752--760, 2011.

Index Terms

• Mathematics of computing

  • Probability and statistics

    • Nonparametric statistics