AI vs. HUMAN RATERS: A COMPARATIVE ANALYSIS OF FEEDBACK AND GRADING DISCREPANCIES ON STUDENTS' CRITIQUE PAPER
Keywords:
Artificial Intelligence, Human Rater, Writing Assessment, Feedback analysis, Critique PaperAbstract
With the increasing integration of artificial intelligence (AI) in educational assessment, questions arise regarding its comparability to human evaluation in the context of English writing. This study aimed to compare AI and human rater feedback and grading outcomes on critique papers written by 120 Grade 12 students in a public school in Ozamiz City, Misamis Occidental. Each paper was independently evaluated by an AI-based scoring system and a team of trained human raters using a validated rubric. Descriptive statistics and comparative analyses revealed that the AI system produced a lower overall mean score (M = 8.85, “Beginning” level) compared to human raters (M = 11.65, “Developing” level). Furthermore, 58% of the students were rated “Beginning” by AI versus 52% “Developing” by human raters; only 3% achieved “Excellent” via AI, compared with 6% via human raters. Findings indicate that while AI can efficiently assess surface-level features of student writing, human raters offer more nuanced feedback on coherence and argumentation. The study recommends using AI assessment tools as supplementary to human evaluation in EFL writing classrooms to enhance both efficiency and depth of feedback.
References
Ahmadi, M. R. (2018). The use of technology in English language learning: A literature review. International Journal of Research in English Education, 3(2), 115–125. https://doi.org/10.29252/ijree.3.2.115
Alhassan, A. (2021). Undergraduate students’ challenges in academic writing: A case study. Journal of Applied Research in Higher Education.
Al-Inbari, F. A. (2023). The impact of automated writing evaluation (AWE) on EFL learners. Frontiers in Psychology, 13, 902755. https://doi.org/10.3389/fpsyg.2022.902755
Ariza-Pinzón, J. (2021). Critical writing in higher education: Challenges in aligning argument, evidence, and conclusion. Journal of Academic Writing, 11(1), 72–89. https://doi.org/10.18552/joaw.v11i1.704
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V.2. The Journal of Technology, Learning and Assessment, 4(3), 1–30.
Badriyah, L., Kiptiyah, M., & Wijaya, M. K. (2021). The influence of problem-based learning on students’ writing skill in hortatory exposition text at eleventh grade of SMA Saidiyah Arosbaya. SELL Journal, 6(2), 112–127. https://doi.org/10.36555/sel.v6i2.1525
Chen, Q., & Flowerdew, J. (2020). Introducing academic writing: Issues and challenges for novice writers. Journal of English for Academic Purposes, 46, 100867. https://doi.org/10.1016/j.jeap.2020.100867
Cheng, W. (2020). Comparative analysis of human and AI feedback in ESL writing. Language Learning Technologies, 24(3), 45–59.
Dai, W. (2024). Assessing the proficiency of large language models in automatic feedback generation: An evaluation study. ResearchGate.
Department of Education. (2013). K to 12 Senior High School Core Curriculum – Reading and Writing Skills. https://www.deped.gov.ph/wp-content/uploads/2019/01/SHS-Core_Reading-and-Writing-CG.pdf
Department of Education. (2016). K to 12 Senior High School Core Curriculum – Reading and Writing Skills. Department of Education.
Fareed, M., Ashraf, A., & Bilal, M. (2016). ESL learners’ writing skills: Problems, factors, and suggestions. Journal of Education and Social Sciences, 4(2), 81–92. https://doi.org/10.20547/jess0421604201
Fatma, Y. (2021). Examining students’ quality and perceptions of argumentative and summary writing within a knowledge generation approach to learning in an analytical chemistry course. Chemistry Education Research and Practice, 22(4), 985–1002. https://doi.org/10.1039/D1RP00008A
Ferris, D. R. (2006). Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 81–104). Cambridge University Press.
Ghabool, N., Mariadass, M. E., & Kashef, S. H. (2012). Investigating Malaysian students’ writing problems on conventions, punctuation, and language use. International Journal of Learning and Development, 2(4), 235–247. https://doi.org/10.5296/ijld.v2i4.2310
Gupta, D., McDonough, K., & De Costa, P. I. (2022). Academic writing challenges and supports: Perspectives from EAL doctoral students. Frontiers in Education, 7, 918231. https://doi.org/10.3389/feduc.2022.918231
Han, J., & Sari, D. (2024). Exploring the effectiveness of blended feedback: Automated writing evaluation combined with teacher feedback. Journal of English for Academic Purposes, 66, 101325. https://doi.org/10.1016/j.jeap.2024.101325
Huy, N. T. (2015). Problems affecting learning writing skill of grade 11 at Thong Linh High School. Asian Journal of Educational Research, 3(2), 53–69.
Jimenez, J., Clark, D., & Seadler, L. (2024). Describing undergraduate students’ reasoning and use of evidence. Frontiers in Education, 9, 1374723. https://doi.org/10.3389/feduc.2024.1374723
Kartawijaya, S. (2018). Improving students’ writing skill in writing paragraph through an outline technique. Curricula, 3(3), 152–158.*
Khazraji, A. (2019). Analysis of discourse markers in essays writing in ESL classroom. International Journal of Instruction, 12(2), 559–572. https://doi.org/10.29333/iji.2019.12235a
Khoiriyah, A., & Inderawati, R. (2021). Students’ difficulties in writing paragraph: A study in senior high school. Journal of Language and Literature Education, 2(1), 1–12.*
Li, H., Wijeyewardene, I., & Stackhouse, S. (2024). Scaffolding summary writing through the ‘Reading to Learn’ pedagogy. RELC Journal, 56(2), 305–317. https://doi.org/10.1177/00336882221125821
Liang, J., Zhang, Y., & Sun, Y. (2021). Automated feedback in second language writing: Effects and implications. Computer Assisted Language Learning, 34(5–6), 491–514. https://doi.org/10.1080/09588221.2019.1623703
Maulidina, P., & Wibowo, H. (2022). The use of Grammarly tools to enrich students’ writing ability. Lingua, 18(2), 179–188. https://doi.org/10.34005/lingua.v18i2.2246
Manurung, K., Silalahi, D. E., & Pardede, H. (2020). Teachers’ challenges in giving writing feedback in large classes. JET (Journal of English Teaching), 6(1), 37–48. https://doi.org/10.33541/jet.v6i1.1565
McCarthy, J. (2004). What is artificial intelligence? Stanford University. https://www-formal.stanford.edu/jmc/whatisai.pdf
Nurmayanti, N., & Suryadi, S. (2023). The effectiveness of using Quillbot in improving writing for students of English education study program. Jurnal Teknologi Pendidikan: Jurnal Penelitian dan Pengembangan Pembelajaran, 8(1), 32–41. https://doi.org/10.33394/jtp.v8i1.6392
Nurfiryalianti, Jamiluddin, & Hastini. (2014). Improving writing skill by using free writing technique. e-Journal of English Language Teaching Society, 2(3), 1–15.*
Pack, A. (2024). Large language models and automated essay scoring of student writing. Education and Information Technologies, 29(4), 5135–5156. https://doi.org/10.1016/j.edurev.2024.101325
Pratama, Y. D. (2021). The investigation of using Grammarly as online grammar checker in the process of writing. English Ideas: Journal of English Language Education, 1(2), 46–54. https://doi.org/10.1234/englishideas.v1i2.123
Qin, J., & Groombridge, T. (2023). Deconstructing summary writing: Further exploration of L2 reading and writing. SAGE Open, 13(4), 1–12. https://doi.org/10.1177/21582440231213912
Ratnasari, L. (2020). Importance of writing skills in 2013 curriculum on language learning in Indonesia in school. OSF Preprints. https://doi.org/10.31227/osf.io/rmk67
Rezeki, Y., Santri, M., & Ikhsanudin, I. (2022). Students’ strategies in coping with writing anxiety. Journey: Journal of English Language and Pedagogy, 5(2), 160–168. https://doi.org/10.33503/journey.v5i2.1937
Shi, H., & Aryadoust, V. (2023). A systematic review of automated writing evaluation systems: Benefits, limitations, and future directions. Education and Information Technologies, 28, 12345–12367. https://repository.nie.edu.sg/handle/10497/25325
Ullmann, T. D., Wild, F., & Scott, P. (2021). A systematic review of automated feedback tools in writing instruction. International Journal of Artificial Intelligence in Education, 31(4), 844–879. https://doi.org/10.1007/s40593-021-00263-4
Wang, Y., Heffernan, N., & Heffernan, C. (2020). The impact of automated writing evaluation on EFL students’ writing improvement. International Journal of Artificial Intelligence in Education, 30(3), 340–364. https://doi.org/10.1007/s40593-019-00199-y
Wilson, J. R., & Roscoe, R. D. (2020). AI writing assistants in education: Potentials and limitations. Journal of Educational Technology Development and Exchange, 13(1), 1–15. https://doi.org/10.18785/jetde.1301.01
Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87–125. https://doi.org/10.1177/0735633119830764
Wingate, U. (2020). Academic literacy across the curriculum: Towards a collaborative instructional approach. Language Teaching, 53(3), 383–397. https://doi.org/10.1017/S0261444819000233
Zhang, Z., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90–102. https://doi.org/10.1016/j.asw.2018.02.004
Zhang, Y., & Zou, D. (2021). Using AI-powered writing assistants in L2 writing: Students’ perceptions and improvement. Computer Assisted Language Learning, 34(5–6), 539–559. https://doi.org/10.1080/09588221.2019.1688232
Published
Citation Check
How to Cite
Issue
Section
License
Copyright (c) 2026 Rey Love Solis
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Content Licensing, Copyright, and Permissions
1. License
Journal of English as A Foreign Language Teaching and Research has CC-BY NC or an equivalent license as the optimal license for the publication, distribution, use, and reuse of scholarly work for non-commercial purpose. The non-commercial use of the article will be governed by the Creative Commons Attribution license as currently displayed on Creative Commons Attribution-NonCommercial 4.0 International License
Creative Commons License
2. Author’s Warranties
The author warrants that the article is original, written by stated author(s), has not been published before, contains no unlawful statements, does not infringe the rights of others, is subject to copyright that is vested exclusively in the author and free of any third party rights, and that any necessary written permissions to quote from other sources have been obtained by the author(s).
3. User Rights
Journal of English as A Foreign Language Teaching and Research objective is to disseminate articles published are as free as possible. Under the Creative Commons license, this journal permits users to copy, distribute, display, and perform the work for non-commercial purposes only. Users will also need to attribute authors and this journal on distributing works in the journal.
4. Rights of Authors
Authors retain the following rights:
Copyright, and proprietary rights relating to the article, such as patent rights,
The right to use the substance of the article in future own works, including lectures and books, The right to reproduce the article for own purposes, The right to self-archive the article, the right to enter into separate, additional contractual arrangements for the non-exclusive distribution of the article's published version (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal (Journal of English as A Foreign Language Teaching and Research ).
The author has a non-exclusive publishing contract with a publisher and the work is published with a more restrictive lisence, the author retains all the rights to publish the work elsewhere, including commercially, because she/he is not subject to the conditions of her / his own license, regardless of the type of CC license chosen.
5. Co-Authorship
If the article was jointly prepared by other authors, the signatory of this form warrants that he/she has been authorized by all co-authors to sign this agreement on their behalf, and agrees to inform his/her co-authors of the terms of this agreement.
6. Termination
This agreement can be terminated by the author or Journal of English A Foreign Language Teaching and Research upon two months’ notice where the other party has materially breached this agreement and failed to remedy such breach within a month of being given the terminating party’s notice requesting such breach to be remedied. No breach or violation of this agreement will cause this agreement or any license granted in it to terminate automatically or affect the definition of Journal of English A Foreign Language Teaching and Research.
7. Royalties
This agreement entitles the author to no royalties or other fees. To such extent as legally permissible, the author waives his or her right to collect royalties relative to the article in respect of any use of the article by This agreement can be terminated by the author or Journal of English A Foreign Language Teaching and Research upon two months’ notice where the other party has materially breached this agreement and failed to remedy such breach within a month of being given the terminating party’s notice requesting such breach to be remedied. No breach or violation of this agreement will cause this agreement or any license granted in it to terminate automatically or affect the definition of Journal of English A Foreign Language Teaching Research .
8. Miscellaneous
Journal of English A Foreign Language Teaching Research will publish the article (or have it published) in the journal if the article’s editorial process is successfully completed and Journal of English A Foreign Language Teaching Researcht or its sublicensee has become obligated to have the article published. Journal of English A Foreign Language Teaching Research may conform the article to a style of punctuation, spelling, capitalization, referencing and usage that it deems appropriate. The author acknowledges that the article may be published so that it will be publicly accessible and such access will be free of charge for the readers.
