Deep Active Learning for Text Classification

ABSTRACT

In recent years, Active Learning (AL) has been applied in the domain of text classification successfully. However, traditional methods need researchers to pay attention to feature extraction of datasets and different features will influence the final accuracy seriously. In this paper, we propose a new method that uses Recurrent Neutral Network (RNN) as the acquisition function in Active Learning called Deep Active Learning (DAL). For DAL, there is no need to consider how to extract features because RNN can use its internal state to process sequences of inputs. We have proved that DAL can achieve the accuracy that cannot be reached by traditional Active Learning methods when dealing with text classification. What's more, DAL can decrease the need of the great number of labeled instances for Deep Learning (DL).

At the same time, we design a strategy to distribute label work to different workers. We have proved by using a proper batch size of instance, we can save much time but not decrease the model's accuracy. Based on this, we provide batch of instances for different workers and the size of batch is determined by worker's ability and scale of dataset, meanwhile, it can be updated with the performance of the workers.

References

1. Aggarwal, C. C., and Zhai, C. 2012. A survey of text classification algorithms. In Mining text data. Springer. 163--222.
2. Chen, Y. and Krause, A. 2013. Near-optimal batch mode active learning and adaptive submodular optimization. In ICML'13 Proceedings of the 30th International Conference on International Conference on Machine Learning.
3. Gal, Y., Islam, R. and Ghahramani, Z. 2016. Deep Bayesian Active Learning with Image Data. In: Advances in Neural Information Processing Systems 29.
4. Gal, Y. 2016. Uncertainty in Deep Learning. Doctor. University of Cambridge.
5. Hassan, S., Rafi, M. and Shaikh, M. 2011. Comparing SVM and naïve Bayes classifiers for text categorization with Wikitology as knowledge enrichment. In IEEE 14th International Multitopic Conference.
6. Hoi, S., Jin R., Zhu J. and Lyu M. 2006. Batch Mode Active Learning and Its Application to Medical Image Classification. In ICML '06 Proceedings of the 23rd international conference on Machine learning.
7. Huang, K. and Lin, H. 2016. A Novel Uncertainty Sampling Algorithm for Cost-sensitive Multiclass Active Learning. In 2016 IEEE 16th International Conference on Data Mining (ICDM).
8. Kim, Y. 2014. Convolutional Neural Networks for Sentence Classification. In Empirical Methods in Natural Language Processing.
9. Lai, S., Xu, L., Liu, K. and Zhao, J. 2015. Recurrent Convolutional Neural Networks for Text Classification. In Twenty-Ninth AAAI Conference on Artificial Intelligence
10. Lilleberg, J., Zhu, Y. and Zhang, Y. 2015. Support Vector Machines and Word2vec for Text Classification with Semantic Features. In IEEE 14th International Conference on Cognitive Informatics & Cognitive Computing.
11. Rajaraman, A., Ullman, J.D. 2011. "Data Mining". Mining of Massive Datasets. pp. 1--17. ISBN 978-1-139-05845-2.
12. Ramos, J.E. 2003. Using TF-IDF to Determine Word Relevance in Document Queries.
13. Settles, B. 2010. Active Learning Literature Survey.
14. Settles, B. and Craven, M. 2008. An Analysis of Active Learning Strategies for Sequence Labeling Tasks. In Empirical Methods in Natural Language Processing.
15. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15 (1929--1958).
16. Sum, M., Li, J., Guo, Z., Zhao, Y., Zheng, Y., Si, X. and Liu, Z. 2016. THUCTC: An Efficient Chinese Text Classifier.
17. Thompson, C., Califf, M. and Mooney, R. 1999. Active Learning for Natural Language Parsing and Information Extraction. In Proceedings of the Sixteenth International Machine Learning Conference.
18. Tong, S. and Koller, D. 2001. Support Vector Machine Active Learning with Applications to Text Classification. Machine Learning Research.
19. Wang, X., Jiang, W. and Luo, Z. 2016. Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts. Osaka, Japan, pp.2428--2437.
20. Yin, W., Kann, K., Yu, M. and Schütze, H. 2017. Comparative Study of CNN and RNN for Natural Language Processing.

Index Terms

• Computing methodologies

  • Machine learning

    • Learning settings

      • Active learning settings