This research addresses the challenge of detecting defects in microscopic images of aluminum inclusions, which are important for the quality and performance of aluminum products among various industries, is dealt with in this investigation. In response to the limited availability of labelled datasets and the intricacy of defect patterns, a new approach that combines Generative Adversarial Networks (GANs) and semantic segmentation techniques is proposed. GANs are employed to generate synthetic images, thereby augmenting the limited dataset, while a VGG16-UNet model is utilized for pixel-level semantic segmentation to accurately detect and classify different types of defects. The study further demonstrates that a fine-tuned VGG16-UNet model, which incorporates improved ground truth images and an inverse class frequency in the loss function, significantly outperforms the baseline model. The fine-tuned model achieved an Intersection over Union (IoU) score of 89%, compared to the baseline model's IoU of 76%. This improvement underscores the effectiveness of the enhancements in ground truth preparation and loss function adjustment. By leveraging transfer learning, data augmentation, and these refinements, the research advances defect detection methods, contributing to enhanced quality control in aluminum manufacturing.