Information Retrieval $($IR$)$ is a field of study dedicated to the organization and retrieval of information from vast and diverse collections of data. In the context of digital information, IR involves the development of techniques and systems that enable users to efficiently access relevant information. Key tasks in IR include indexing, searching, and ranking documents based on their relevance to user queries.

This assignment delves into various aspects of Information Retrieval, exploring fundamental concepts and practical applications. Throughout the assignment, we will examine techniques for document indexing, retrieval models, and the evaluation of IR systems using metrics such as precision, recall, and mean average precision. By delving into these topics, we aim to gain a comprehensive understanding of the core principles that govern effective information retrieval.

In this lab, you are going to implement a standard document processing pipeline and then build a simple search engine based on it:

$-$ Building an inverted index,

$-$ Answering queries using this index with various retrieval models.

$-$ Evaluate the models implemented.

This assignment is based on a subset of the TREC $2003$ Web Topic Distillation will be employed, featuring a curated subset of the WT$10$g $($Web Track $10$ gigabyte$)$ corpus as the document collection. This chosen dataset will serve as the foundation for rigorously assessing the performance of the implemented Information Retrieval models. Utilizing a well$-$established benchmark such as TREC ensures a standardized and meaningful evaluation, allowing for robust comparisons between different models.

You can locate the necessary data in the following locations:

$-**$Data Folder:$**$ 'government$\_$data'

$-**$Documents Folder:$**$ 'documents'

$-**$Topics $($Queries$)$ File:$**$ 'gov.topics'

$-*$Format:$*$ Each line follows the pattern $``$

$-**$Qrels $($Query Relevance Judgements$)$ File:$**$ 'gov.qrels'

$-*$Format:$*$ Each line follows the pattern $`0`$

$-``$: The unique identifier for the query.

$-`$Q$0`$: A literal indicating the query and is usually constant.

$-``$: The identifier of the retrieved document.

$-``$: The rank assigned to the document by the retrieval system.

$-``$: The retrieval score assigned to the document by the system.

$-``$: A unique identifier for the run or retrieval system.