Högskolan i Skövde

The retail sector is undergoing an apparent digital transformation that completely revolutionises shopping operations. To stay competitive, retailer stakeholders are forced to rethink and improve their business models to provide an attractive personalised experience to consumers. The self-service checkout process is at the heart of this transformation and should be designed to identify the products accurately and detect any possible anomalous behaviour. In this paper, we introduce a product verification system based on OCR classification and Mondrian conformal prediction. The proposed system includes three components: OCR reading, text classification and product verification. By using image data from existing grocery stores, the system can detect anomalies with high performance, even when there is partial text information on the products. This makes the system applicable for reducing shrinkage loss (caused, for example, by employee theft or shoplifting) in grocery stores by identifying fraudulent behaviours such as barcode switching and miss-scan. Additionally, OCR reading with NLP classification shows that it is in itself a powerful classifier of products. 

Self‐checkout systems aim to provide a seamless and high-quality shopping experience and increase the profitability of stores. These advantages come with some challenges such as shrinkage loss. To overcome these challenges, automatic recognition of the purchased products is a potential solution. In this context, one of the big issues that emerge is the data shifting, which is caused by the difference between the environment in which the recognition model is trained and the environment in which the model is deployed. In this paper, we use transfer learning to handle the shift caused by the change of camera and lens or their position as well as critical factors, mainly lighting, reflection, and occlusion. We motivate the use of Natural Language Processing (NLP) techniques on textual data extracted from images instead of using image recognition to study the efficiency of transfer learning techniques. The results show that cross-domain NLP retail recognition using the BERT language model only results in a small reduction in performance between the source and target domain. Furthermore, a small number of additional training samples from the target domain improves the model to perform comparable as a model trained on the source domain.

Grocery product recognition techniques are emerging in the retail sector and are used to provide automatic checkout counters, reduce self-checkout fraud, and support inventory management. However, recognizing grocery products using machine learning models is challenging due to the vast number of products, their similarities, and changes in appearance. To address these challenges, more complex models are created by adding additional modalities, such as text from product packages. But these complex models pose additional challenges in terms of model interpretability. Machine learning experts and system developers need tools and techniques conveying interpretations to enable the evaluation and improvement of multimodal production recognition models. In this work, we propose thus an approach to provide local and global explanations that allow us to assess multimodal models for product recognition. We evaluate this approach on a large fine-grained grocery product dataset captured from a real-world environment. To assess the utility of our approach, experiments are conducted for three types of multimodal models. The results show that our approach provides fine-grained local explanations while being able to aggregate those into global explanations for each type of product. In addition, we observe a disparity between different multimodal models, in what type of features they learn and what modality each model focuses on. This provides valuable insight to further improve the accuracy and robustness of multimodal product recognition models for grocery product recognition.

Automatic recognition of grocery products can be used to improve customer flow at checkouts and reduce labor costs and store losses. Product recognition is, however, a challenging task for machine learning-based solutions due to the large number of products and their variations in appearance. In this work, we tackle the challenge of fine-grained product recognition by first extracting a large dataset from a grocery store containing products that are only differentiable by subtle details. Then, we propose a multimodal product recognition approach that uses product images with extracted OCR text from packages to improve fine-grained recognition of grocery products. We evaluate several image and text models separately and then combine them using different multimodal models of varying complexities. The results show that image and textual information complement each other in multimodal models and enable a classifier with greater recognition performance than unimodal models, especially when the number of training samples is limited. Therefore, this approach is suitable for many different scenarios in which product recognition is used to further improve recognition performance. The dataset can be found at https://github.com/Tubbias/finegrainocr.

Automatic checkout systems are designed to predict a complete shopping receipt using an image from the checkout area. These systems require high classification accuracy across numerous classes and must operate in real-time, despite domain differences between training data and real-world conditions. Building on recent advancements, we propose a method that outperforms current solutions and can be applied in real-time in automatic checkout systems. Our method leverages the Segment Anything Model to extract high-quality masks from lab product images, which are then transformed into synthetic checkout images and adapted to the real domain using contrastive unpaired translation. We train a product recognition model with data augmentation, named SCA+Y8, and further improve it through fine-tuning with pseudo-labels from unlabeled checkout images, resulting in an improved model called SCAFT+Y8. SCAFT+Y8 achieves a great increase in state-of-the-art performance, with an average receipt classification accuracy of 97.58%, and shows strong performance in smaller models, indicating the potential for deployment on low-cost edge devices.