Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 = nonrelevant, 1 = relevant). Let us assume that youve written an IR system that for this query returns the set of documents {4, 5, 6, 7, 8}.

1.Calculate precision, recall, and F1 of your system if a document is considered relevant only if the two judges agree.

2.Calculate precision, recall, and F1 of your system if a document is considered relevant if either judge thinks it is relevant.

Explain the answers please (where did we get the numbers from)

Answers:

1.

P = 1/5 R = 1/2

F1 = 2PR / (P+R) = 2 * 1/5 * 1/2 / (1/5 + 1/2) = 2/7 = 0.285714

2.

P=1

R = 5/10

F = 2 * 1 * 1/2 / (1 + 1/2) = 2/3 = 0.666666

DocID 2 34 56 78 9 10 11 12 Judge 1 0 0 1 000 0 Judge 200 0 0 00 1