Refining the prediction of user satisfaction on chat-based AI applications with unsupervised filtering of rating text inconsistencies

Abstract

The swift development of artificial intelligence (AI) technology has triggered substantial changes, particularly evident in the emergence of chat-based services driven by large language models. With the increasing number of users utilizing these services, understanding and analysing user satisfaction becomes crucial for service improvement. While previous studies have explored leveraging online reviews as indicators of user satisfaction, efficiently collecting and analysing extensive datasets remain a challenge. This research aims to address this challenge by proposing a framework to handle extensive review datasets from the Google Play Store, employing natural language processing with machine learning techniques for sentiment analysis. Specifically, the authors collect review data of chat-based AI applications and perform filtering through majority voting of multiple unsupervised sentiment analyses. This framework is a proposed methodology for eliminating inconsistencies between ratings and contents. Subsequently, the authors conduct supervised sentiment analysis using various machine learning and deep learning algorithms. The experimental results confirm the effectiveness of the proposed approach showing improvement in prediction accuracy with cost efficiency. In summary, the findings of this study enhance the predictive performance of user satisfaction for improving service quality in chat-based AI applications and provide valuable insights for the advancement of next-generation chat-based AI services.