Converting programs to visual representation by reading the compiled binary

Gathering visual context the hard way (but it always works).

Problem

Gathering the visual representation of a mobile application is non-trivial. Code draws the interface elements into what the user sees. Ultimately an app is composed of machine language that a CPU executes. This patent covers a technique that can work on any app to create the user viewable screens that can be used by a translator to provide contextual translations.

Solution

The compiled code is analyzed and converted into a visual representation by looking at the binary commands that paint the screen and using that data to recreate a visual representation of the app which is rendered in a browser.

Impact

The beauty of this method is that it works on any source code; Objective-C, Swift, Unity, LiveCode, C#, Java, etc. Since everything must be converted to machine language, this technique works no matter what language or tool the code was created in.

This is the generic method for converting code into visual context. The easier method, when possible, is to intercept screen draws.

Status

US Patent Application Number 61928918
Filed 20-JAN-2015