Högskolan i Skövde

This study investigates the use of machine learning (ML) models to distinguish between Largue Language Model generated (LLM-generated) and student-written programming assignments. It compares the CSEDM and LLM-generated datasets from ChatGPT, DeepSeek, and Qwen to identify distinguishing features. The LLM dataset has been generated with several prompt engineering techniques to ensure a diverse and representative series of outputs. Several ML models, including XGBoost, SVM, and LightGBM, are evaluated for classification performance. This study has shown that Deepseek is the most difficult to classify LLM, whereas the ML models excelled in different areas, with no one model being preeminent. The study also explores prompt engineering techniques to generate LLM code resembling student submissions. Results include an analysis of what ML models perform the best and why, visualising what differences LLM software artifacts have to student artifacts. Furthermore, the results emphasize and visualize features importance for different models using feature extraction. The findings offer promising results and insights into educational policy and the development of stronger automated detection tools.