UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision

Chun-Fu (Richard) Chen, Marco Pistoia, Conglei Shi, Paolo Girolami, Joseph W. Ligman, Yong Wang

Workflow of proposed UI testing system, UI X-RAY


User Interface/eXperience (UI/UX) significantly affects the lifetime of any software program, particularly mobile apps. A bad UX can undermine the success of a mobile app even if that app enables sophisticated capabilities. A good UX, however, needs to be supported of a highly functional and user friendly UI design.In spite of the importance of building mobile apps based on solid UI designs, UI discrepancies—inconsistencies between UI design and implementation—are among the most numerous and expensive defects encountered during testing.This paper presents UI X-Ray, an interactive UI testing system that integrates computer-vision methods to facilitate the correction of UI discrepancies—such as inconsistent positions, sizes and colors of objects and fonts.Using UI X-Ray does not require any programming experience; therefore, UI X-Ray can be used even by non-programmers—particularly designers—which significantly reduces the overhead involved in writing tests.With the feature of interactive interface, UI testers can quickly generate defect reports and revision instructions—which would otherwise be done manually. We verified our UI X-Ray on 4 developed mobile apps of which the entire development history was saved. UI X-Ray achieved a 99.03\% true-positive rate, which significantly surpassed the 20.92% true-positive rate obtained via manual analysis. Furthermore, evaluating the results of our automated analysis can be completed quickly (< 1 minute per view on average) compared to hours of manual work required by UI testers.On the other hand, UI X-Ray received the appreciations from skilled designers and UI X-Ray improves their current work flow to generate UI defect reports and revision instructions.The proposed system, UI X-Ray, presented in this paper has recently become part of a commercial product.


| pdf |


Chun-Fu (Richard) Chen, Marco Pistoia, Conglei Shi, Paolo Girolami, Joseph W. Ligman, Yong Wang."UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision". In Proceedings of the 22nd International Conference on Intelligent User Interfaces