Högskolan i Skövde

Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for enhancing the factuality and contextual relevance of large language model (LLM) outputs, particularly in enterprise applications such as help-desk automation. This thesis evaluates three retrieval strategies—sparse (BM25), dense (SBERT + FAISS), and hybrid (Reciprocal Rank Fusion)—combined with two open-source LLMs, Mistral-7B-Instruct and LLaMA-2-7B-Chat, to assess their performance in a domain-specific customer support scenario. Using the Doc2Dial dataset, we implement a RAG pipeline and assess model outputs using both automated metrics (Hit@K, nDCG@K, ROUGE-L, BERTScore, Cosine Similarity) and human evaluations across multiple dimensions (factuality, relevance, clarity, and completeness). Results show that dense and hybrid retrievers outperform sparse retrieval in both retrieval and generation quality, and Mistral-7B exhibits slightly stronger performance in human-rated clarity and completeness. A Pearson correlation analysis reveals that ROUGE-L aligns more closely with human judgment compared to other automated metrics. The thesis highlights the limitations of relying solely on automatic metrics and underscores the importance of qualitative human evaluation in enterprise-oriented RAG systems. Future work can explore more diverse domains, retrieval architectures, and fine-tuning strategies to further improve response quality and contextual grounding.