publications

2026

1. IST

   FairST: A novel approach for machine learning bias repair through latent sensitive attribute translation

   Carmen Meinson, Max Hort, and Federica Sarro

   Information and Software Technology, 2026

   Abs DOI

   Context: As Machine Learning (ML) models are increasingly used in critical decision-making software, concerns have been raised about these systems perpetuating or exacerbating existing historical biases. Consequently, there has been a growing research interest in developing methods to test for fairness and repair biases in ML software, particularly for legally protected attributes like gender, age, race. Objectives: In this work, we set out to repair bias for both single and multiple protected attributes (a.k.a. intersectional fairness) of pre-trained machine learning models. Methods: We propose a novel model- and task-agnostic debiasing method, Fair Subgroup Translation (FairST), based on fair representation learning via auto-encoders. To the best of our knowledge, this is the first approach based on the principle of Fair Representation Learning devised for post-processing bias repair. Results: We empirically evaluate the effectiveness of using FairST to repair a pre-trained Neural Network model used for seven classification tasks involving both single and multiple protected attributes, and benchmark its performance with state-of-the-art fairness repair methods (i.e., Learning Fair Representations, Reweighing, FairBalance and FairMask). We also investigate if the effectiveness of FairST varies when using it to repair bias of other popular ML models (namely Logistic Regression, Support Vector Machine, Gaussian Naive Bayes, Decision Tree and Random Forest). Conclusion: The results demonstrate that FairST consistently achieves superior single and intersectional fairness with respect to all benchmarking methods for all classification tasks considered in our empirical study. This supports the potential of using FairST for ML bias repair, and opens up a rich agenda of future work including its application to repair bias arising in tasks of a different nature such as multi-class or image-based problems.

2025

1. TOSEM

   Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances

   Zhao Tian, Minghua Ma, Max Hort, and 3 more authors

   ACM Trans. Softw. Eng. Methodol., Jun 2025

   Just Accepted

   Abs DOI

   Fairness testing aims at mitigating unintended discrimination in the decision-making process of data-driven AI systems. Individual discrimination may occur when an AI model makes different decisions for two distinct individuals who are distinguishable solely according to protected attributes, such as age and race. Such instances reveal biased AI behaviour, and are called Individual Discriminatory Instances (IDIs).In this paper, we propose an approach for the selection of the initial seeds to generate IDIs for fairness testing. Previous studies mainly used random initial seeds to this end. However this phase is crucial, as these seeds are the basis of the follow-up IDIs generation. We dubbed our proposed seed selection approach I&D. It generates a large number of initial IDIs exhibiting a great diversity, aiming at improving the overall performance of fairness testing.Our empirical study reveals that I&D is able to produce a larger number of IDIs with respect to four state-of-the-art IDI generation approaches, generating 1.86X more IDIs on average. When using the IDIs generated with I&D for retraining a machine learning model, the percentage of IDIs in the input space (mathbbI) is decreased by 24.9% on average, implying that I&D is effective for improving the model’s fairness.

2. preprint

   Large Language Models: A Survey of Surveys

   Max Hort, Fernando Vallecillos-Ruiz, and Leon Moonen

   Nov 2025

   working paper or preprint

   PDF
3. preprint

   Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models: A Reflection

   David Williams, Max Hort, Maria Kechagia, and 3 more authors

   Nov 2025
4. preprint

   Wisdom and Delusion of LLM Ensembles for Code Generation and Repair

   Fernando Vallecillos-Ruiz, Max Hort, and Leon Moonen

   Nov 2025
5. preprint

   The Impact of Fine-tuning Large Language Models on Automated Program Repair

   Roman Macháček, Anastasiia Grishina, Max Hort, and 1 more author

   Nov 2025
6. EASE

   The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models

   Fernando Vallecillos Ruiz, Max Hort, and Leon Moonen

   In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, , Nov 2025

   Abs DOI

   Automatic program repair (APR) aims at reducing the manual efforts required to identify and fix errors in source code. Before the rise of Large Language Model (LLM)-based agents, a common strategy was simply to increase the number of generated patches, sometimes to the thousands, which usually yielded better repair results on benchmarks. More recently, self-iterative capabilities enabled LLMs to refine patches over multiple rounds guided by feedback. However, literature often focuses on many iterations and disregards different numbers of outputs.We investigate an APR pipeline that balances these two approaches, the generation of multiple outputs and multiple rounds of iteration, while imposing a limit of 10 total patches per bug. We apply three SOTA instruction-tuned LLMs – DeepSeekCoder-Instruct, Codellama-Instruct, Llama3.1-Instruct – to the APR task. We further fine-tune each model on an APR dataset with three sizes (1K, 30K, 65K) and two techniques (Full Fine-Tuning and LoRA), allowing us to assess their repair capabilities on two APR benchmarks: HumanEval-Java and Defects4J.Our results show that by using only a fraction (<1%) of the fine-tuning dataset, we can achieve improvements of up to 78% in the number of plausible patches generated, challenging prior studies that reported limited gains using Full Fine-Tuning. However, we find that exceeding certain thresholds leads to diminishing outcomes, likely due to overfitting. Moreover, we show that base models greatly benefit from creating patches in an iterative fashion rather than generating them all at once. In addition, the benefit of iterative strategies becomes more pronounced in complex benchmarks. Even fine-tuned models, while benefiting less from iterations, still gain advantages, particularly on complex benchmarks. The research underscores the need for balanced APR strategies that combine multi-output generation and iterative refinement.

7. ICST

   Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection

   Max Hort, Linas Vidziunas, and Leon Moonen

   In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Nov 2025

   DOI
8. ICST

   Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces

   Max Hort and Leon Moonen

   In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), Nov 2025

   DOI
9. TOSEM

   Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation

   Fernando Vallecillos Ruiz, Anastasiia Grishina, Max Hort, and 1 more author

   ACM Trans. Softw. Eng. Methodol., Oct 2025

   Just Accepted

   Abs DOI

   Research shows that errors in natural language can be corrected by translating texts to another language and back using language models. We explore to what extent this latent correction capability extends to Automated Program Repair (APR) by investigating Round-Trip Translation (RTT): translating code from one programming language into another programming or natural language and back, using Large Language Models (LLMs). We hypothesize that RTT restores patterns most commonly seen in the LLM’s training corpora through regression toward the mean, replacing infrequent bugs with more frequent, natural, bug-free code. To test this hypothesis, we employ nine LLMs and four common APR benchmarks in Java, and perform a detailed quantitative and qualitative analysis of RTT-generated patches. We find that RTT through English generates plausible patches for 100 of 164 bugs with GPT-4 on the HumanEval-Java benchmark, and 97 are found to be correct in our manual assessment. Moreover, RTT uniquely generates plausible patches for 46 bugs that were missed by LLMs specifically fine-tuned for APR. While this demonstrates the viability of RTT for APR, we also observe limitations, such as a lower overall bug fix rate than the state-of-the-art and diluting the original coding style. We analyze the impact of these limitations and discuss the potential of using RTT as a complementary component in APR frameworks.

2024

1. ESEM

   A Comparative Study on Large Language Models for Log Parsing

   Merve Astekin, Max Hort, and Leon Moonen

   In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Barcelona, Spain, Oct 2024

   Abs DOI

   Background: Log messages provide valuable information about the status of software systems. This information is provided in an unstructured fashion and automated approaches are applied to extract relevant parameters. To ease this process, log parsing can be applied, which transforms log messages into structured log templates. Recent advances in language models have led to several studies that apply ChatGPT to the task of log parsing with promising results. However, the performance of other state-of-the-art large language models (LLMs) on the log parsing task remains unclear. Aims: In this study, we investigate the current capability of state-of-the-art LLMs to perform log parsing. Method: We select six recent LLMs, including both paid proprietary (GPT-3.5, Claude 2.1) and four free-to-use open models, and compare their performance on system logs obtained from a selection of mature open-source projects. We design two different prompting approaches and apply the LLMs on 1,354 log templates across 16 different projects. We evaluate their effectiveness, in the number of correctly identified templates, and the syntactic similarity between the generated templates and the ground truth. Results: We found that free-to-use models are able to compete with paid models, with CodeLlama extracting 10% more log templates correctly than GPT-3.5. Moreover, we provide qualitative insights into how usable these six models are for log parsing. Conclusions: Our results reveal that some of the smaller, free-to-use LLMs can considerably assist log parsing compared to their paid proprietary competitors, especially code-specialized models.

2. JRC

   Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey

   Max Hort, Zhenpeng Chen, Jie M. Zhang, and 2 more authors

   ACM J. Responsib. Comput., Jun 2024

   Abs DOI

   This article provides a comprehensive survey of bias mitigation methods for achieving fairness in Machine Learning (ML) models. We collect a total of 341 publications concerning bias mitigation for ML classifiers. These methods can be distinguished based on their intervention procedure (i.e., pre-processing, in-processing, post-processing) and the technique they apply. We investigate how existing bias mitigation methods are evaluated in the literature. In particular, we consider datasets, metrics, and benchmarking. Based on the gathered insights (e.g., What is the most popular fairness metric? How many datasets are used for evaluating bias mitigation methods?), we hope to support practitioners in making informed choices when developing and evaluating new bias mitigation methods.

3. InteNSE

   An Exploratory Study on How Non-Determinism in Large Language Models Affects Log Parsing

   Merve Astekin, Max Hort, and Leon Moonen

   In Proceedings of the ACM/IEEE 2nd International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, Lisbon, Portugal, Jun 2024

   Abs DOI

   Most software systems used in production generate system logs that provide a rich source of information about the status and execution behavior of the system. These logs are commonly used to ensure the reliability and maintainability of software systems. The first step toward automated log analysis is generally log parsing, which aims to transform unstructured log messages into structured log templates and extract the corresponding parameters.Recently, Large Language Models (LLMs) such as ChatGPT have shown promising results on a wide range of software engineering tasks, including log parsing. However, the extent to which non-determinism influences log parsing using LLMs remains unclear. In particular, it is important to investigate whether LLMs behave consistently when faced with the same log message multiple times.In this study, we investigate the impact of non-determinism in state-of-the-art LLMs while performing log parsing. Specifically, we select six LLMs, including both paid proprietary and free-to-use models, and evaluate their non-determinism on 16 system logs obtained from a selection of mature open-source projects. The results of our study reveal varying degrees of non-determinism among models. Moreover, they show that there is no guarantee for deterministic results even with a temperature of zero.

4. TOSEM

   Fairness Testing: A Comprehensive Survey and Analysis of Trends

   Zhenpeng Chen, Jie M. Zhang, Max Hort, and 2 more authors

   ACM Trans. Softw. Eng. Methodol., Jun 2024

   Abs DOI

   Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this article offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely adopted datasets and open-source tools for fairness testing.

5. EMSE

   Search-based automatic repair for fairness and accuracy in decision-making software

   Max Hort, Jie M Zhang, Federica Sarro, and 1 more author

   Empirical Software Engineering, Jun 2024

2023

1. FSE

   The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification

   Anastasiia Grishina, Max Hort, and Leon Moonen

   In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, San Francisco, CA, USA, Jun 2023

   Abs DOI PDF

   The use of modern Natural Language Processing (NLP) techniques has shown to be beneficial for software engineering tasks, such as vulnerability detection and type inference. However, training deep NLP models requires significant computational resources. This paper explores techniques that aim at achieving the best usage of resources and available information in these models. We propose a generic approach, EarlyBIRD, to build composite representations of code from the early layers of a pre-trained transformer model. We empirically investigate the viability of this approach on the CodeBERT model by comparing the performance of 12 strategies for creating composite representations with the standard practice of only using the last encoder layer. Our evaluation on four datasets shows that several early layer combinations yield better performance on defect detection, and some combinations improve multi-class classification. More specifically, we obtain a +2 average improvement of detection accuracy on Devign with only 3 out of 12 layers of CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early layers can be used to obtain better results using the same resources, as well as to reduce resource usage during fine-tuning and inference.

2. ESEM

   An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code

   Max Hort, Anastasiia Grishina, and Leon Moonen

   In 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Jun 2023

   DOI PDF
3. GECCO

   Multi-objective Search for Gender-fair and Semantically Correct Word Embeddings (HOP GECCO’23)

   Max Hort, Rebecca Moussa, and Federica Sarro

   In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, Lisbon, Portugal, Jun 2023

   Abs DOI

   Mitigating algorithmic bias during the development life cycle of AI-enabled software is crucial given that any bias in these algorithms is inherited by the software systems using them. At the Hot-off-the-Press GECCO track, we aim at disseminating our article Multi-objective search for gender-fair and semantically correct word embeddings. Applied Soft Computing, 2023 [5]. In this work, we exploit multi-objective search to strike an optimal balance between reducing gender bias and improving semantic correctness of word embedding models, which are at the core of many AI-enabled systems. Our results show that, while single-objective search approaches are able to reduce the gender bias of word embeddings, they also reduce their semantic correctness. On the other hand, multi-objective approaches are successful in improving both goals, in contrast to existing work which solely focuses on reducing gender bias. Our results show that multi-objective evolutionary approaches can be successfully exploited to address bias in AI-enable software systems, and we encourage the research community to further explore opportunities in this direction.

4. ASC

   Multi-objective search for gender-fair and semantically correct word embeddings

   Max Hort, Rebecca Moussa, and Federica Sarro

   Applied Soft Computing, Jun 2023

   Abs DOI PDF

   Fairness is a crucial non-functional requirement of modern software systems that rely on the use of Artificial Intelligence (AI) to make decisions regarding our daily lives in application domains such as justice, healthcare and education. In fact, these algorithms can exhibit unwanted discriminatory behaviours that create unfair outcomes when the software is used, such as giving privilege to one group of users over another (e.g., males vs. females). Mitigating algorithmic bias during the development life cycle of AI-enabled software is crucial given that any bias in these algorithms is inherited by the software systems using them. However, previous work has shown that mitigating bias can impact the performance of such systems. Therefore, we propose herein a novel use of soft computing for improving AI-enabled software fairness. Specifically, we exploit multi-objective search, as opposed to previous work optimising fairness only, to strike an optimal balance between reducing gender bias and improving semantic correctness of word embedding models, which are at the core of many AI-enabled systems. To assess the effectiveness of our proposal, we carry out a thorough empirical study based on the most recent best practice for the evaluation of search-based approaches and AI-enabled software. We explore seven different search-based approaches, and benchmark them against both baseline and state-of-the-art approaches applied to a popular and widely used word embedding model, namely Word2Vec. Our results show that multi-objective search outperforms single-objective search, and generates word embeddings that are strictly better than the original ones in both objectives, bias and semantic correctness, for all investigated cases. Additionally, our approach generates word embeddings of higher semantic correctness than those generated by using state-of-the-art techniques in all cases, while also achieving a higher degree of fairness in 67% of the cases. These findings show the feasibility and effectiveness of multi-objective search as a tool for engineers to incorporate fair and accurate word embedding models in their AI-enabled systems.

2022

1. GECCO

   Py2Cy: a genetic improvement tool to speed up python

   James Zhong, Max Hort, and Federica Sarro

   In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Boston, Massachusetts, Jun 2022

   Abs DOI PDF

   Due to its ease of use and wide range of custom libraries, Python has quickly gained popularity and is used by a wide range of developers all over the world. While Python allows for fast writing of source code, the resulting programs are slow to execute when compared to programs written in other programming languages like C. One of the reasons for its slow execution time is the dynamic typing of variables. Cython is an extension to Python, which can achieve execution speed-ups by compiler optimization. One possibility for improvements is the use of static typing, which can be added to Python scripts by developers. To alleviate the need for manual effort, we create Py2Cy, a Genetic Improvement tool for automatically converting Python scripts to statically typed Cython scripts. To show the feasibility of improving runtime with Py2Cy, we optimize a Python script for generating Fibonacci numbers. The results show that Py2Cy is able to speed up the execution time by up to a factor of 18.

2. GeBNLP

   An Empirical Study on the Fairness of Pre-trained Word Embeddings

   Emeralda Sesari, Max Hort, and Federica Sarro

   In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), Jul 2022

   Abs DOI PDF

   Pre-trained word embedding models are easily distributed and applied, as they alleviate users from the effort to train models themselves. With widely distributed models, it is important to ensure that they do not exhibit undesired behaviour, such as biases against population groups. For this purpose, we carry out an empirical study on evaluating the bias of 15 publicly available, pre-trained word embeddings model based on three training algorithms (GloVe, word2vec, and fastText) with regard to four bias metrics (WEAT, SEMBIAS,DIRECT BIAS, and ECT). The choice of word embedding models and bias metrics is motivated by a literature survey over 37 publications which quantified bias on pre-trained word embeddings. Our results indicate that fastText is the least biased model (in 8 out of 12 cases) and small vector lengths lead to a higher bias.

3. FairWare

   Privileged and unprivileged groups: an empirical study on the impact of the age attribute on fairness

   Max Hort and Federica Sarro

   In Proceedings of the 2nd International Workshop on Equitable Data and Technology, Pittsburgh, Pennsylvania, Jul 2022

   Abs DOI PDF

   Recent advances in software fairness investigate bias in the treatment of different population groups, which are devised based on attributes such as gender, race and age. Groups are divided into privileged groups (favourable treatment) and unprivileged groups (unfavourable treatment). To truthfully represent the real world and to measure the degree of bias according to age (young vs. old), one needs to pick a threshold to separate those groups.In this study we investigate two popular datasets (i.e., German and Bank) and the bias observed when using every possible age threshold in order to divide the population into "young" and "old" groups, in combination with three different Machine Learning models (i.e., Logistic Regression, Decision Tree, Support Vector Machine). Our results show that age thresholds do not only impact the intensity of bias in these datasets, but also the direction (i.e., which population group receives a favourable outcome). For the two investigated datasets, we present a selection of suitable age thresholds. We also found strong and very strong correlations between the dataset bias and the respective bias of trained classification models, in 83% of the cases studied.

4. TSE

   A Survey of Performance Optimization for Mobile Applications

   Max Hort, Maria Kechagia, Federica Sarro, and 1 more author

   IEEE Transactions on Software Engineering, Jul 2022

   DOI PDF

2021

1. ASE

   Did You Do Your Homework? Raising Awareness on Software Fairness and Discrimination

   Max Hort and Federica Sarro

   In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Jul 2021

   DOI PDF
2. FSE

   Fairea: a model behaviour mutation approach to benchmarking bias mitigation methods

   Max Hort, Jie M. Zhang, Federica Sarro, and 1 more author

   In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, Jul 2021

   Abs DOI PDF

   The increasingly wide uptake of Machine Learning (ML) has raised the significance of the problem of tackling bias (i.e., unfairness), making it a primary software engineering concern. In this paper, we introduce Fairea, a model behaviour mutation approach to benchmarking ML bias mitigation methods. We also report on a large-scale empirical study to test the effectiveness of 12 widely-studied bias mitigation methods. Our results reveal that, surprisingly, bias mitigation methods have a poor effectiveness in 49% of the cases. In particular, 15% of the mitigation cases have worse fairness-accuracy trade-offs than the baseline established by Fairea; 34% of the cases have a decrease in accuracy and an increase in bias. Fairea has been made publicly available for software engineers and researchers to evaluate their bias mitigation methods.

3. GECCO

   The effect of offspring population size on NSGA-II: a preliminary study

   Max Hort and Federica Sarro

   In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Lille, France, Jul 2021

   Abs DOI PDF

   Non-Dominated Sorting Genetic Algorithm (NSGA-II) is one of the most popular Multi-Objective Evolutionary Algorithms (MOEA) and has been applied to a large range of problems.Previous studies have shown that parameter tuning can improve NSGA-II performance. However, the tuning of the offspring population size, which guides the exploration-exploitation trade-off in NSGA-II, has been overlooked so far. Previous work has generally used the population size as the default offspring population size for NSGA-II.We therefore investigate the impact of offspring population size on the performance of NSGA-II. We carry out an empirical study by comparing the effectiveness of three configurations vs. the default NSGA-II configuration on six optimization problems based on four Pareto front quality indicators and statistical tests.Our findings show that the performance of NSGA-II can be improved by reducing the offspring population size and in turn increasing the number of generations. This leads to similar or statistically significant better results than those obtained by using the default NSGA-II configuration in 92% of the experiments performed.

2020

1. GECCO

   Optimising word embeddings with search-based approaches

   Max Hort and Federica Sarro

   In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, Cancún, Mexico, Jul 2020

   Abs DOI PDF

   Word embeddings have rapidly become an all-purpose tool for a diverse range of real world applications. This development is nurtured by the availability and applicability of pre-trained models. However, their usage faces the risk of being inaccurate when used in domains different from the ones they were trained on.In this paper, we formulate the adaptation of word embeddings as a vector multiplication problem, which enables us to apply search methods to explore potential word embedding adaptations with respect to their semantic correctness. To assess the effectiveness of our proposal, we empirically investigate the use of both local and global search-based approaches (i.e. Hill Climbing, Tabu Search and Genetic Algorithm) in order to maximise the semantic correctness of a popular Word2Vec pre-trained model (namely GoogleNews) when applied to another domain (i.e. the MEN dataset).The results of our study reveal that Hill Climbing, Tabu Search and Genetic Algorithm perform equally well and all outperform the original GoogleNews model as well as a baseline model based on Random Search. This shows that optimising word embeddings with search-based approaches is possible and effective.