Högskolan i Skövde

Knowledge is a key strategic resource for organizations, and an organization’s ability to successfully manage and transfer knowledge has a wide range of impacts on key performance indicators. This study looked at how Elektroautomatik, a small Swedish automation company could improve their knowledge management process through use of generative AI. Prior to this study employees at Elektroautomatik used a OneNote document (Kunskapsbank) to manage knowledge across various automation projects. 

This study proposed solution in the form of an LLM-powered chatbot that utilized a Neo4J GraphRAG database built off the Kunskapsbank document. An iterative design and creation process was used to develop the chatbot first utilizing a RAG retrieval model, then a GraphRAG with a monolingual query function for the chatbot, and finally a bilingual query function that could search information in the GraphRAG in both English and Swedish. Developing the RAG and GraphRAG required specialized data generation and query handling processes as the Kunskapsbank document contained a mix of information in a mixed media format of text, images, and tables, and was written in both English and Swedish. 

The chatbot was evaluated based on a set test questions and ground truths developed by Elektroautomatik using the RAGAS metrics context precision, context recall, semantic similarity, factual correctness, and faithfulness. The three methods, RAG, monolingual GraphRAG query, and bilingual GraphRAG query were all evaluated, and the bilingual GraphRAG query had the best performance overall. Through the development of this LLM-powered chatbot the study was able to demonstrate how generative AI could be used to develop a knowledge management system that could handle a mixed media and bilingual knowledge dataset. 

Future studies should focus on improving performance of Swedish language queries, evaluating the system on a more robust set of test questions, and following up on user testing evaluation after completing a full application. Additionally, the LLM-powered chatbot knowledge management system developed in this thesis should be tested at other manufacturing automation companies to verify results.