RANKED INFORMATION RETRIEVAL USING WEIGHTED TF IDF7

Abstract

Ranked Information Retrieval using Weighted TF IDF Document Retrieval is the task of retrieving a relevant Document in response to a query, a question, or a reference Document. Tasks such as question answering, summarization, novelty detection, and information provenance make use of a Document retrieval module as a preprocessing step. The performance of these systems is dependent on the quality of the Document‐retrieval module. Other tasks such as information extraction and machine translation operate on Documents, either using them as training data, or as the unit of input or output (or both), and may benefit from Document retrieval to build a training corpus, or as a post‐processing step. In this thesis we begin by studying IR Model, then we build a through understanding of exiting IR algorithms like TFIDF, Okapi BM25 and Pivoted length normalization to name a few. During the study of the mentioned algorithms we come up with some deficiencies in retrieval algorithms and started working to eradicate those deficiencies. We proposed a better approach for scoring documents named Weighted TF IDF (WTF IDF) instead of TF IDF where terms are counted rather than weighted with respect to locality of documents and term order. More over we planned to cope with different writing styles by looking for synonym query along with original query, this increase the chances of retrieving some novel information from the corpus. We have provided the implementation of exiting algorithms and compare the performance with proposed approach WTF IDF and presented the result. The proposed approach has better results than the exiting ones