Automated Annotation of Test Types using LLMs: A Multi-LLM Workflow for Machine Learning Test-Related Commits
1st International Workshop on Agentic AI for Next-Generation Software Development, AgenticDev 2026, Munich, Almanya, 12 - 16 Ekim 2026, ss.166-167, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1145/3843282.3844433
- Basıldığı Şehir: Munich
- Basıldığı Ülke: Almanya
- Sayfa Sayıları: ss.166-167
- Anahtar Kelimeler: Empirical Evaluation, Large Language Models, Machine Learning Testing, Test Type Annotation
- Galatasaray Üniversitesi Adresli: Evet
Özet
Test-type annotation classifies commits based on the type of software test introduced or modified. While there are established taxonomies for testing machine learning (ML) software systems, implementing these manually at the commit level can be costly. This makes large language models (LLMs) a promising alternative. Utilizing an eight-category test-type taxonomy, we evaluate 97 manually labeled commits from scikit-learn using GPT-4o, Claude Sonnet, and Qwen 2.5-7B-Instruct-Turbo LLMs through four prompting strategies. This analysis focused on their classification performance, stability, agreement with experts, top-2 candidate recovery, and multi-LLM voting workflows (S1-S4) along with a corresponding analysis on Keras. Notably, GPT-4o in a zero-shot setting demonstrated the highest stability, and permitting a second label led to a reduction in single-prediction errors (e.g., Claude CoT-ZS Hit@1 = [59.79]%, Hit@2 = [77.32]%). Our most effective voting strategy (S1) surpassed the solo baseline, with its superior macro-F1 score (S1 > S2 > S3 > S4) also holding for Keras.