Item Response Theory for Natural Language Processing

John P. Lalor, Pedro Rodriguez, João Sedoc, Jose Hernandez-Orallo


Abstract
This tutorial will introduce the NLP community to Item Response Theory (IRT; Baker 2001). IRT is a method from the field of psychometrics for model and dataset assessment. IRT has been used for decades to build test sets for human subjects and estimate latent characteristics of dataset examples. Recently, there has been an uptick in work applying IRT to tasks in NLP. It is our goal to introduce the wider NLP community to IRT and show its benefits for a number of NLP tasks. From this tutorial, we hope to encourage wider adoption of IRT among NLP researchers.
Anthology ID:
2024.eacl-tutorials.2
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Mohsen Mesgar, Sharid Loáiciga
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9–13
Language:
URL:
https://aclanthology.org/2024.eacl-tutorials.2
DOI:
Bibkey:
Cite (ACL):
John P. Lalor, Pedro Rodriguez, João Sedoc, and Jose Hernandez-Orallo. 2024. Item Response Theory for Natural Language Processing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 9–13, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Item Response Theory for Natural Language Processing (Lalor et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-tutorials.2.pdf
Video:
 https://aclanthology.org/2024.eacl-tutorials.2.mp4