MultiFusion: A boosting approach for multimedia fusion

Abstract

The multimodal data usually contain complementary, correlated and redundant information. Thus, multimodal fusion is useful for many multisensor applications. Here, a novel multimodal fusion algorithm is proposed, which is referred to as “MultiFusion.” The approach adopts a boosting structure where the atomic event is considered as the fusion unit. The correlation of multimodal data is used to form an overall classifier in each iteration. Moreover, by adopting the Adaboost-like structure, the overall fusion performance is improved. Both the simulation experiment and the real application show the effectiveness of the MultiFusion approach. Our approach can be applied in different multimodal applications to exploit the multimedia data characteristics and improve the performance.

References

1. Aly, R. and Hiemstra, D. 2009. Concept detectors: how good is good enough? In Procedings of the ACM International Conference on Multimedia. 233--242.
2. Amores, J., Sebe, N., Radeva, P., Gevers, T., and Smeulders, A. W. M. 2004. Boosting contextual information in content-based image retrieval. In Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval. 31--38.
3. Atrey, P. K., Kankanhalli, M. S., and Jain, R. 2006. Information assimilation framework for event detection in multimedia surveillance systems. Multimedia Syst. 12, 3, 239--253.
4. Barbu, C., Iqbal, R. T., and Peng, J. 2005. Classifier fusion using shared sampling distribution for boosting. In Proceedings of the International Conference on Data Mining. 34--41.
5. Contreras, F. M., Urunuela, C. O., and del Rincon, J. M. 2009. Adaboost multiple feature selection and combination for face recognition. In Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis. 338--345.
6. Dasarathy, B. V. 1994. Decision Fusion. Computer Society Press.
7. Freund and Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. J. Compu. Syst. Sci. 55.
8. Freund, Y. and Schapire, R. E. 1996. Experiments with a new boosting algorithm. In Proceedings of the International Conference on Machine Learning. 148--156.
9. Guo, G. and Zhang, H. 2001. Boosting for fast face recognition. Tech. rep. MSR-TR-2001-16, Microsoft Research.
10. Guo, W., Qi, Q., Jiang, L., and Zhang, P. 2008. A new method of sar image target recognition based on adaboost algorithm. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium.
11. Iyengar, G., Nock, H., and Neti, C. 2003. Audio-visual synchrony for detection of monologues in video archives. In Proceedings of the IEEE International Conference on Multimedia and Expo 1, 329--332.
12. Joshi, D., Naphade, M. R., and Natsev, A. 2007. Semantics reinforcement and fusion learning for multimedia streams. In Proceedings of the International Conference on Image and Video Retrieval. 309--316.
13. Maghooli, K. and Moin, M. S. 2004. A new approach on multimodal biometrics based on combining neural networks using adaboost. In Biometric Authentication. Lecture Notes in Computer Science Vol. 3087 332--341.
14. Parikh, D., Kim, M. T., Oagaro, J., Mandayam, S., and Polikar, R. 2004. Combining classifiers for multisensor data fusion. In Proceedings of the International Conference on Systems, Man and Cybernetics. 1232--1237.
15. Pickering, M. J., Ruger, S. M., and Sinclair, D. 2002. Video retrieval by feature learning in key frames. In Proceedings of the International Conference on Image and Video Retrieval. 309--317.
16. Polikar, R. 2006. Ensemble based systems in decision making. IEEE Circu. Syst. Maga. 6, 3, 21--45.
17. Polikar, R., Upda, L., Upda, S. S., and Honavar, V. 2001. Learn++: an incremental learning algorithm for supervised neural networks. IEEE Trans. Syst. Man, Cyber. Part C 31, 4, 497--508.
18. Schapire, R. E. 1990. The strength of weak learnability. Mach. Learn. 5, 2.
19. Tieu, K. and Viola, P. 2004. Boosting image retrieval. Int. J. Comput. Vis. 56, 1--2, 17--36.
20. Xue, F. and Ding, X. 2006. 3D+2D face localization using boosting in multi-modal feature space. In Proceedings of the International Conference on Pattern Recognition. III: 499--502.
21. Zhang, L., Li, M. J., and Zhang, H. J. 2002. Boosting image orientation detection with indoor vs. outdoor classification. In Proceedings of the IEEE Workshop on Applications of Computer Vision. 95--99.
22. Zhang, T. and Yu, B. 2005. Boosting with early stopping: Convergence and consistency. Ann. Stat. 33, 1538--1579.

Index Terms

• Computing methodologies

  • Machine learning

• Information systems

  • Information systems applications

    • Multimedia information systems