The Web & Software Engineering research group at HTWK Leipzig has made it onto the podium in all four categories of the international BEA competition for the evaluation of German short answers
The Web & Software Engineering (WSE Research) researchgroup at HTWK Leipzig secured a place on the podium in all four categories at the international BEA competition in San Diego, which assessed German short answers. Competing against nine teams, PhD student Jonas Gwozdz and Prof. Dr Andreas Both secured second place three times and third place once in July 2026. The task combines natural language processing with a question from everyday educational practice: how well can an AI recognise whether a pupil has answered a subject-specific question correctly?
The competition forms part of the ‘Building Educational Applications’ workshop – BEA for short – which was held as part of one of the world’s leading conferences on Natural Language Processing (NLP) and computational linguistics: ACL 2026. In a competition known as a ‘shared task’, research teams compare the methods they have developed using the same data and evaluation criteria.
Factually correct, partly correct or incorrect?
The background to this competition is the research objective of achieving a high-quality automatic assessment of answers to examination questions. A particular problem here is that those who formulate an answer in their own words rarely use the exact terms found in a model answer. A different phrasing can convey the same content; conversely, an answer that sounds technically sound may fail to capture a key connection. This makes automatic marking a challenging task. A simple comparison of individual words is not sufficient for this purpose.
In the BEA competition, therefore, all participating systems were provided not only with the question and the pupil’s answer but also with a detailed marking scheme. This describes which elements are required for a correct or partially correct answer. The examples are taken from German-language questions in the fields of mathematics and natural sciences. A total of 7,899 previously assessed answers to 78 questions were available for training and development.
The four evaluation criteria test different requirements. One part examines new answers to questions that are already known, whilst another examines answers to questions that were previously unknown. In addition, there are two assessment schemes: either ‘correct’, ‘partially correct’ and ‘incorrect’, or a binary decision in which even partially correct answers are counted as incorrect. Particularly in the case of unknown questions, the system must be able to apply a new assessment criterion.
When is it beneficial to combine several procedures?
The WSE team investigated various methods of evaluation. Language models were given specific instructions and appropriate, pre-evaluated example responses. In addition, the researchers adapted so-called large language models (LLMs) to the task through additional training. To do this, they used LoRA, a method that trains only a comparatively small set of additional model parameters.
For familiar questions, it proved helpful to combine the assessments of several models on a weighted basis. For unfamiliar questions, however, a single, larger, fine-tuned model performed better than the combination. This distinction is relevant to the research: a good solution for recurring tasks cannot be transferred to new questions without further testing.
The work was carried out in the research group led by Prof. Both at the Faculty of Computer Science and Media. PhD student Jonas Gwozdz is investigating there how language models can support the assessment of examination performance. The competition provides a comparison under defined conditions. Whether the results can be transferred to other subjects and levels of education must be investigated separately. Integrating the findings into teachers’ work also remains a separate research task.
Publication:
Jonas Gwozdz and Andreas Both. 2026.WSE Research at BEA 2026 Shared Task 2: Multi-Strategy Rubric-Based Short Answer Scoring for German. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), pages 1210–1216, San Diego, California, USA. Association for Computational Linguistics.
