Collaborative Human-AI Risk Annotation: Co-Annotating Online Incivility with CHAIRA
Clicks: 32
ID: 283244
2024
Article Quality & Performance Metrics
Overall Quality
Not rated
Combines reader engagement with the AI quality analysis. This
article has not been analysed, so there is no overall score —
reader engagement is measured and shown alongside.
Reader Engagement
Emerging Content
9.3
/100
32 views
16 readers
AI Quality Assessment
Not analyzed
Readership in this journal
EmergingRanked #699 of 803 articles by views in arXiv
Most read
Least read
Bar heights use a square-root scale. Only the 120 most-read articles are drawn; the journal has 803 in total.
Mint this article as an NFT
Not yet mintedCreate a permanent, verifiable on-chain record of this article on the Scimatic Network. The NFT is held in your Journament account, and you can withdraw it to your own wallet at any time.
5
SUSD
one-off · no wallet required
Abstract
Collaborative human-AI annotation is a promising approach for various tasks
with large-scale and complex data. Tools and methods to support effective
human-AI collaboration for data annotation are an important direction for
research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk
Annotation tool that enables human and AI agents to collaboratively annotate
online incivility. We leveraged Large Language Models (LLMs) to facilitate the
interaction between human and AI annotators and examine four different
prompting strategies. The developed CHAIRA system combines multiple prompting
approaches with human-AI collaboration for online incivility data annotation.
We evaluated CHAIRA on 457 user comments with ground truth labels based on the
inter-rater agreement between human and AI coders. We found that the most
collaborative prompt supported a high level of agreement between a human agent
and AI, comparable to that of two human coders. While the AI missed some
implicit incivility that human coders easily identified, it also spotted
politically nuanced incivility that human coders overlooked. Our study reveals
the benefits and challenges of using AI agents for incivility annotation and
provides design implications and best practices for human-AI collaboration in
subjective data annotation.
| Reference Key |
singh2024collaborative
Use this key to autocite in the manuscript while using
SciMatic Manuscript Manager or Thesis Manager
|
|---|---|
| Authors | Jinkyung Katie Park; Rahul Dev Ellezhuthil; Pamela Wisniewski; Vivek Singh |
| Journal | arXiv |
| Year | 2024 |
| DOI |
DOI not found
|
| URL | |
| Keywords |
Citations
No citations found. To add a citation, contact the admin at info@scimatic.org
Comments
No comments yet. Be the first to comment on this article.