Japanese-English Sentence Translation Exercises Dataset for Automatic Grading

Naoki Miura, Hiroaki Funayama, Seiya Kikuchi, Yuichiroh Matsubayashi, Yuya Iwase, Kentaro Inui


Abstract
This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs), that have been used in the early stage of L2 language learning.We formalize the task as grading student responses for each rubric criterion pre-specified by the educators.We then create a dataset for STE between Japanese and English including 21 questions, along with a total of 3,498 student responses (167 on average).The answer responses were collected from students and crowd workers.Using this dataset, we demonstrate the performance of baselines including a finetuned BERT model and GPT-3.5 with few-shot learning. Experimental results showed that the baseline model with fine-tuned BERT was able to classify correct responses with approximately 90% in F1, but only less than 80% for incorrect responses. Furthermore, GPT-3.5 with few-shot learning shows a poorer result than the BERT model, indicating that our newly proposed task presents a challenging issue, even for the state-of-the-art large language model.
Anthology ID:
2024.eacl-srw.21
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Neele Falk, Sara Papi, Mike Zhang
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
266–278
Language:
URL:
https://aclanthology.org/2024.eacl-srw.21
DOI:
Bibkey:
Cite (ACL):
Naoki Miura, Hiroaki Funayama, Seiya Kikuchi, Yuichiroh Matsubayashi, Yuya Iwase, and Kentaro Inui. 2024. Japanese-English Sentence Translation Exercises Dataset for Automatic Grading. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 266–278, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Japanese-English Sentence Translation Exercises Dataset for Automatic Grading (Miura et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-srw.21.pdf
Video:
 https://aclanthology.org/2024.eacl-srw.21.mp4