Converting programs to visual representation by intercepting screen draws

Gathering visual context the easy way.

Problem

Gathering the visual representation of an application’s screens is non-trivial. Some programming languages have commands that take a set of defined graphic elements and paint them onto the screen. By intercepting those commands, one can capture the definition of the graphic elements, and relay those definitions to a system that converts them into a visual representation of what the user sees on screen.

Solution

This covers a technique where a code library in the app watches for screen draws on apps running in a simulator. As the developer is developing their app and running it in a simulator, the code library captures what the screen looks like and where the strings and images are displayed. This can work for all programming environments that include screen draw callbacks.

Impact

It is far easier and accurate to capture screen definitions using this technique than to decompile the compiled binary.

Status

US Patent Application Number 61928927
Filed 20-JAN-2015