Deep Recurrent Conditional Random Field Network for Protein Secondary Prediction

ABSTRACT

Deep learning has become the state-of-the-art method for predicting protein secondary structure from only its amino acid residues and sequence profile. Building upon these results, we propose to combine a bi-directional recurrent neural network (biRNN) with a conditional random field (CRF), which we call the biRNN-CRF. The biRNN-CRF may be seen as an improved alternative to an auto-regressive uni-directional RNN where predictions are performed sequentially conditioning on the prediction in the previous time-step. The CRF is instead nearest neighbor-aware and models for the joint distribution of the labels for all time-steps. We condition the CRF on the output of biRNN, which learns a distributed representation based on the entire sequence. The biRNN-CRF is therefore close to ideally suited for the secondary structure task because a high degree of cross-talk between neighboring elements can be expected. We validate the model on several benchmark datasets. For example, on CB513, a model with 1.7 million parameters, achieves a Q8 accuracy of 69.4 for single model and 70.9 for ensemble, which to our knowledge is state-of-the-art.

References

1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Gordon Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. 2016. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. CoRR abs/1603.04467 (2016). https://arxiv.org/abs/1603.04467 Software available from tensorflow.org.
2. D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos, E. Elsen, J. Engel, L. Fan, C. Fougner, T. Han, A. Hannun, B. Jun, P. LeGresley, L. Lin, S. Narang, A. Ng, S. Ozair, R. Prenger, J. Raiman, S. Satheesh, D. Seetapun, S. Sengupta, Y. Wang, Z. Wang, C. Wang, B. Xiao, D. Yogatama, J. Zhan, and Z. Zhu. 2015. Deep Speech 2: End-to- End Speech Recognition in English and Mandarin. CoRR abs/1512.02595 (2015). http://arxiv.org/abs/1512.02595
3. J. Ba, J. Kiros, and G. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
4. D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473 (2014). https://arxiv.org/abs/1409.0473
5. C. Bishop. 2006. Pattern recognition and machine learning. Springer.
6. A. Busia, J. Collins, and N. Jaitly. 2016. Protein Secondary Structure Prediction Using Deep Multi-scale Convolutional Neural Networks and Next-Step Conditioning. CoRR abs/1611.01503 (2016). http://arxiv.org/abs/1611.01503
7. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).
8. F. Chollet. 2015. Keras: Theano-based deep learning library. (2015). https://github.com/fchollet/keras
9. J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555 (2014). http://arxiv.org/abs/1412.3555
10. X. Glorot, A. Bordes, and Y. Bengio. 2011. Deep Sparse Rectifier Neural Networks. In Aistats, Vol. 15. 275.
11. A. Graves. 2012. Supervised sequence labelling with recurrent neural networks. Springer (2012).
12. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Deep Residual Learning for Image Recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385
13. S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (Nov. 1997), 1735--1780.
14. D. Jones. 1999. Protein secondary structure prediction based on position-specific scoring matrices. Journal of Molecular Biology 292, 2 (1999), 195 -- 202.
15. D. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representation.
16. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
17. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML '01). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 282--289. http://dl.acm.org/citation.cfm?id=645530.655813
18. Y. LeCun, Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444.
19. Z. Li and Y. Yu. 2016. Protein Secondary Structure Prediction Using Cascaded Convolutional and Recurrent Neural Networks. arXiv preprint arXiv:1604.07176 (2016).
20. B. Monastyrskyy, A. Kryshtafovych, J. Moult, A. Tramontano, and K. Fidelis. 2014. Assessment of protein disorder region predictions in CASP10. Proteins: Structure, Function, and Bioinformatics 82, S2 (2014), 127--137.
21. J. Moult, K. Fidelis, A. Kryshtafovych, T. Schwede, and A. Tramontano. 2014. Critical assessment of methods of protein structure prediction (CASP)round x. Proteins: Structure, Function, and Bioinformatics 82, S2 (2014), 1--6.
22. D. Ruck, S. Rogers, M. Kabrisky, M. Oxley, and B. Suter. 1990. The multilayer perceptron as an approximation to a Bayes optimal discriminant function. Neural Networks, IEEE Transactions on 1, 4 (1990), 296--298.
23. M. Schuster and K. Paliwal. 1997. Bidirectional Recurrent Neural Networks. Trans. Sig. Proc. 45, 11 (Nov. 1997), 2673--2681.
24. M. Singh. 2005. Predicting protein secondary and supersecondary structure. Handbook of Computational Molecular Biology, hapman & Hall CRC Computer and Information Science Series (2005).
25. S. Sønderby, C. Sønderby, H. Nielsen, and O. Winther. 2015. Convolutional LSTM networks for subcellular localization of proteins. In International Conference on Algorithms for Computational Biology. Springer, 68--80.
26. S. Sønderby and O. Winther. 2014. Protein secondary structure prediction with long short term memory networks. arXiv preprint arXiv:1412.7828 (2014).
27. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. (2014). http://dl.acm.org/citation.cfm?id=2627435.2670313
28. A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. 2016. WaveNet: A Generative Model for Raw Audio. CoRR abs/1609.03499 (2016). http://arxiv.org/abs/1609.03499
29. S. Wang, J. Peng, J. Ma, and J. Xu. 2016. Protein secondary structure prediction using deep convolutional neural fields. Scientific reports 6 (2016).
30. . Wu, M. Schuster, Z. Chen, Q. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, and others. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 (2016). https://arxiv.org/abs/1609.08144
31. J. Zhou and O. Troyanskaya. 2014. Deep Supervised and Convolutional Generative Stochastic Network for Protein Secondary Structure Prediction. In ICML. 745--753.

Index Terms

• Computing methodologies

  • Machine learning

    • Learning paradigms

      • Supervised learning

        • Supervised learning by classification

    • Learning settings

      • Batch learning

    • Machine learning approaches

      • Neural networks