Use this link to cite:

https://hdl.handle.net/2183/48046

Hybrid LLM and Sentence Transformer Evaluation for ITS Programming Exercises

Thumbnail Image

Files

PerezSanchez_Beatriz_2025_Hybrid_LLM_and_Sentence_Transformer_Evaluation.pdf (508.48 KB)

Identifiers

URI: https://hdl.handle.net/2183/48046
DOI: 10.1109/ICETC66579.2025.11387358

Publication date

2025

Authors

Ortega Morla, Javier
Ferreiro Villamor, Daniel

Advisors

Other responsabilities

Journal Title

Bibliographic citation

J. Ortega-Morla, D. Ferreiro, A. Paz-Lopez, B. Pérez-Sánchez and S. Guerreiro-Santalla, "Hybrid LLM and Sentence Transformer Evaluation for ITS Programming Exercises," 2025 International Conference on Education Technology and Computers (ICETC), Barcelona, Spain, 2025, pp. 221-225, doi: 10.1109/ICETC66579.2025.11387358.

Type of academic work

Academic degree

Abstract

[Abstract]: This paper explores automatic evaluation and feedback generation for varied student output in programming exercises. To this end, a hybrid approach was implemented, combining Sentence Transformer Models for output classification with General Purpose Large Language Models for generating brief, informative feedback. The solution was tested on a dataset of 1,300 manually labeled student outputs from programming exams at the University of A Coruña. Finally, this system will be integrated into ProgTutor, an Intelligent Tutoring System, to replace the current evaluation method and enable its assessment in large-scale, real-world conditions.

Description

© 2025 IEEE. This version of the article has been accepted for publication, after peer review. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The Version of Record is available online at: https://doi.org/10.1109/ICETC66579.2025.11387358 Presented at: 2025 International Conference on Education Technology and Computers (ICETC), 18-21 September 2025, Barcelona, Spain

Keywords

Intelligent Tutoring Systems ITS Large Language Models LLM Automatic Evaluation

Editor version

https://doi.org/10.1109/ICETC66579.2025.11387358

Rights

Copyright © 2025, IEEE

Collections

Investigación (FIC)