2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2015)
Lincoln, NE, USA
Nov. 9, 2015 to Nov. 13, 2015
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ASE.2015.32
When developing the user interface code of a mobile application, in practice a big gap exists between the digital conceptual drawings of graphic artists and working user interface code. Currently, programmers bridge this gap manually, by reimplementing the conceptual drawings in code, which is cumbersome and expensive. To bridge this gap, we introduce the first technique to automatically Reverse Engineer Mobile Application User Interfaces (REMAUI). On a given input bitmap REMAUI identifies user interface elements such as images, texts, containers, and lists, via computer vision and optical character recognition (OCR) techniques. In our experiments on 488 screenshots of over 100 popular third-party Android and iOS applications, REMAUI-generated user interfaces were similar to the originals, both pixel-by-pixel and in terms of their runtime user interface hierarchies. REMAUI's average overall runtime on a standard desktop computer was 9 seconds.
Graphical user interfaces, Mobile applications, Smart phones, Layout, Optical character recognition software, Containers
T. A. Nguyen and C. Csallner, "Reverse Engineering Mobile Application User Interfaces with REMAUI (T)," 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), Lincoln, NE, USA, 2015, pp. 248-259.