Artificial Intelligence (AI) has revolutionized mobile apps across various domains, from productivity to entertainment. However, understanding how users perceive and interact with these AI-driven features remains a crucial yet underexplored aspect. To bridge this gap, we conducted a comprehensive large-scale study on user feedback for AI-powered mobile apps.

Our research leveraged a curated dataset of 292 AI-driven apps across 14 categories, featuring over 894K AI-specific reviews from Google Play. We developed and validated a multi-stage analysis pipeline that begins with human-labeled benchmarking and systematically evaluates large language models (LLMs) and prompting strategies.

Each stage, including review classification, aspect-sentiment extraction, and clustering, was thoroughly validated for accuracy and consistency. This pipeline enabled scalable and high-precision analysis of user feedback, extracting over one million aspect-sentiment pairs clustered into 18 positive and 15 negative user topics.

Our analysis revealed that users consistently focus on a narrow set of themes: positive comments emphasize the value of AI-powered apps in enhancing productivity, reliability, and personalized assistance. Conversely, negative feedback highlights technical issues (e.g., scanning and recognition), pricing concerns, and limitations in language support.

What's more, our pipeline surfaced both satisfaction with one feature and frustration with another within the same review. These fine-grained, co-occurring sentiments are often missed by traditional approaches that treat positive and negative feedback in isolation or rely on coarse-grained analysis.

Our approach provides a more faithful reflection of real-world user experiences with AI-powered apps. Furthermore, category-aware analysis uncovers both universal drivers of satisfaction and domain-specific frustrations.

Discovering the Hidden Patterns

By analyzing user feedback for AI-powered mobile apps, we uncovered key themes that shape user perception:

  • Productivity and reliability: Users praise AI-powered apps for streamlining tasks, improving efficiency, and ensuring consistent performance.
  • Personalized assistance: Users appreciate AI-driven features that offer tailored support, reducing the need for manual intervention.
  • Technical issues: Users criticize AI-powered apps for technical glitches, such as scanning and recognition problems, which hinder overall experience.

The Power of Fine-Grained Analysis

Our pipeline's ability to extract fine-grained sentiment pairs reveals a more nuanced understanding of user experiences. By examining co-occurring sentiments within the same review, we can identify:

  • Dual feedback: Users may express satisfaction with one feature while criticizing another.
  • Contextual relevance: User feedback often references specific features or scenarios.

Unlocking AI-Powered Mobile Apps' Potential

Our research highlights the importance of user-centric design and development for AI-powered mobile apps. By incorporating insights from this study, app developers can:

  • Improve user experience by addressing technical issues and enhancing reliability.
  • Develop more personalized and efficient features that cater to users' needs.

By unlocking the secrets of user feedback for AI-powered mobile apps, we take a significant step towards creating more effective and user-centric experiences.