Agreement, the F-Measure, and Reliability in Information Retrieval

Clicks: 1
ID: 301086
2005
Free PDF
Article Quality & Performance Metrics
Overall Quality
Not rated
Combines reader engagement with the AI quality analysis. This article has not been analysed, so there is no overall score — reader engagement is measured and shown alongside.
AI Quality Assessment
Not analyzed
Readership in this journal

Ranked #72 of 76 articles by views in Journal of the American Medical Informatics Association : JAMIA

Most read Least read

Bar heights use a square-root scale.

Mint this article as an NFT
Not yet minted

Create a permanent, verifiable on-chain record of this article on the Scimatic Network. The NFT is held in your Journament account, and you can withdraw it to your own wallet at any time.

5 SUSD one-off · no wallet required
Sign in to mint
Abstract
Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the κ statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that κ approaches these measures as the number of negative cases grows large. Positive specific agreement—or the equivalent F-measure—may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.
Reference Key
openalex_W79139011 Use this key to autocite in the manuscript while using SciMatic Manuscript Manager or Thesis Manager
Authors George Hripcsak
Journal Journal of the American Medical Informatics Association : JAMIA
Year 2005
DOI
10.1197/jamia.m1733
URL
https://academic.oup.com/jamia/article-pdf/12/3/296/2429751/12-3-296.pdf
Keywords Keywords not found

Citations

No citations found. To add a citation, contact the admin at info@scimatic.org

Comments

No comments yet. Be the first to comment on this article.