Text document clustering based on frequent word sequences

ABSTRACT

In this paper, we propose a new text clustering algorithm, named Clustering based on Frequent Word Sequences (CFWS). A word sequence is frequent if it occurs in more than certain percentage of the documents in the text database. In the past, the vector space model was commonly used for information retrieval, but it treats documents as bags of words, ignoring the sequential pattern of word occurrences in the documents. However, the meaning of natural languages strongly depends on the word sequences, and the frequent word sequences can provide compact and valuable information about the text database. Bisecting k-means and FIHC algorithms are evaluated on the performance of text clustering, and are compared with the proposed CFWS algorithm. It has been shown that CFWS has much better performance.

REFERENCES

• 1: B. C. M. Fung, K. Wang, and M. Ester, "Hierarchical Document Clustering Using Frequent Itemsets," Proc. of SIAM Int'l Conf. on Data Mining, 2003.
• 2: High Accuracy Retrieval from Documents (HARD) Track of Text Retrieval Conference, 2004.
• 3: The Lemur Toolkit for Language Modeling and Information Retrieval, http://www-2.cs.cmu.edu/~lemur/.
• 4: M. Steinbach, G. Karypis, and V. Kumar, "A Comparison of Document Clustering Techniques," KDD-2000 Workshop on Text Mining, 2000.
• 5: P. Weiner, "Linear Pattern Matching Algorithms," Proc. of the 14th Annual Symp. on Foundation of Computer Science, 1973, pp. 1--11.
• 6: Oren Zamir , Oren Etzioni, Web document clustering: a feasibility demonstration, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.46-54, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290956]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering

General Terms:
Algorithms, Performance

Keywords:
clustering, frequent word sequences, text documents, web search