Rendering Flight Telemetry in Platform-Independent 3D Touch-Interactive Simulations
Matthew A. Noyes
AUG 23, 2012 23:44 PM
A+ A A-

Three dimensional graphics are a critical asset to engineers of all disciplines. Graphics play many roles in product creation, from conception and prototyping to test data visualization. Many modern computer-assisted simulations use the same mouse-controlled 2D graphical user interfaces developed decades ago.

To explore new innovations in 3D graphics for engineering, a new data-driven simulation mechanism was created. This system uses flight telemetry to model vehicle state as a three-dimensional simulation for both standard platforms (PC and Mac) and iOS mobile devices (iPhones, iPads, and iPod touches). The data-delivery system is compatible with any mission using the Integrated Test and Operations System (ITOS) developed at Goddard Space Flight Center, with only slight modifications to the renderer required.

The system uses a Lightstreamer server to deliver telemetry to a wide variety of devices, which uses the UNITY game engine to translate these data into visual state. These core technologies, ITOS, Lightstreamer, and UNITY, are largely commercial-based solutions developed in C, Javascript, and .NET C#. They are modular and decoupled, requiring only slight modification for each mission. They allow engineers to monitor flight status in the field on extremely configurable mobile devices. Two NASA applications on the app store use derivatives of this technology—iMorpheus and ISSLive.

The iMorpheus application, which models state of the NASA Project Morpheus lander during flight tests, uses the ITOS tool to beam telemetry to a communication center near the launch site.

The communication center forwards ITOS telemetry to the Mission Control Center, where it is republished via a Lightstreamer data adapter. Lightstreamer is a push service, refreshing live data as soon as new values are available. If data are not updated in a single refresh cycle, they are not forwarded to the client, reducing network bandwidth. This makes Lightstreamer ideal for scalable data delivery.

Lightstreamer provides clients for many different software environments, including .NET, Java, and HTML/Javascript. Due to clients' reliability on libraries unsupported by UNITY, a custom interface was implemented for iMorpheus until Lightstreamer releases a full UNITY client.

Separate sockets are used for data streaming and control events. If a stream connection dies, it can be reestablished (bound) with a session ID. An iMorpheus session opens a stream connection in a background thread to the Lightstreamer server, where it obtains and stores a session ID to reconnect in the event of stream closure. Lightstreamer automatically closes the stream connection once the number of bytes sent to the client exceeds a server-specified content length.

In a real-time telemetry delivery scenario, this should be avoided as it will produce a highly observable latency. The iMorpheus client requests a content length equal to the maximum integer constant; in the event of disconnect, the application will reopen a stream connection using the stored session ID.

Following stream socket initialization, the application opens a control connection and submits an HTTP request over a TCP socket, which adds a subscription to all telemetry variables required for modeling state, then closes the control connection. The application will begin receiving raw bytes, which are stored in a fixed-size client-side buffer. Thread progress halts until data are received. No timeout is specified to prevent disconnection in the absence of real-time updates.

The Morpheus vehicle provides updates at 10 Hz; while Lightstreamer's maximum update frequency is 1 Hz. To compensate for this latency, the server delivers sets of telemetry values as JSON strings containing an Arraylist of 10 states each. The iMorpheus application parses this object and updates vehicle state over the course of a second. iMorpheus vehicle state will have a minimum latency of 1 second from data delivery; there will be additional lag derived from linearly interpolating the spacecraft along designated waypoints.

When data are received, they are decoded into an ASCII text string containing unicode characters for quotation marks, carriage returns and line feeds. An explicit String.Replace call translates the data into well-formatted JSON. This string begins with a comma-delimited numeric sequence indicating the table number and item number of the update, followed by a vertical-pipe delimited sequence of JSON strings, each specifying a single cell in the subscribed table. Each cell corresponds to a single telemetry variable.

The data is decoded by MiniJSON into a Hashtable. Individual values are pulled from this Hashtable and placed into Telemetry objects, each storing data for a single state variable. A Telemetry objects array representing a single Lightstreamer response is placed on a thread-safe Queue. This Queue is simultaneously accessed by the main thread to populate vehicle state.

Due to the magnitude of data required to model vehicle state, as well as the significant overhead of using JSON strings, data are split into packets of unknown size. Each time data are read and decoded, the number of bytes received is stored and the buffer size is updated accordingly. If a Lightstreamer table header is located in the middle of the buffer, multiple datasets have been received; the application will then store the first complete set of data in a string for parsing; the remaining bytes are copied to the beginning of the buffer and the buffer size is adjusted accordingly. Bytes stored beyond the index specified by the buffer size are not zeroed, but are instead ignored by the application. If the application has not detected complete reception of data, it waits until more bytes are received before checking again.

After complete reception of bytes from the stream connection, but before parsing, the application also checks the data string for a LOOP message. Lightstreamer sends status messages if necessary in the final cell of a table request. A LOOP request is sent upon reaching content length; if received, the application opens a new stream connection which intelligently restores communication with the stored session ID.

After splitting the table along vertical pipes and whitespace into the individual table items, the items are ready to be parsed from the received bytes. All connection code is managed by an abstract superclass, hiding implementation details from the software engineer.

Data are parsed using the parseBytes method defined by the Lightstreamer client. The parseBytes method takes individual light-streamer table cells, decodes the JSON strings into values and assembles the values into telemetry objects, where each telemetry object represents a single state variable. State updates are performed from the main thread, which grabs a reference to (but does not dequeue) the next telemetry update.

If the Queue has telemetry available, the program creates a set of a local replacement variables for each piece of data to be updated. Telemetry values are fetched from a helper function, GetTelemetryValue(), which takes an ITOS telemetry identifier string, a queue, and the old telemetry value. If a telemetry object with the specified name parameter exists in the head of the Queue, its stored value is returned; otherwise, the old variable value is returned.

Since Lightstreamer is change-only, new state telemetry may not contain all possible variables; it is simple to assume that in a given frame, all variables will remain constant if no updates are available for them.

Reducing the GUI interface to a single draw call will greatly improve performance on older, mobile hardware. Researching more features congruent with engineering analysis, such as higher-fidelity vehicle models with interior mechanisms, telemetry to model state changes in those mechanisms, and graphical feedback for failure events such as applying a blinking warning texture.

Clicking or touching the affected part could display more information about the problem cause from the telemetry perspective. Also, the ability to show or hide spacecraft layers (circuits, fuel lines, life support sensors, etc.) so that different engineering teams may focus on different subsets of the telemetry, would better utilize the groundwork already in place for this application. The infrastructure has already been implemented for any telemetry transmitted from the vehicle, all that remains is a comparable client-side mapping. Finally, adding more detail and structures to the terrain allows for a more realistic and immersive simulation.

Three dimensional graphics displays and touch-based interfaces represent initial steps toward intuitive human-computer interaction by allowing users to manipulate virtual environments in a manner analogous to everyday human experience. The less a user must learn to fully utilize system capabilities, the more intuitive the system. This application serves to represent complicated flight telemetry as an interactive simulation using real-time data, recorded data, and local user control, with the ability to forward this information to many devices simultaneously.

This software product has applications in both engineering analysis and public outreach due to its ease of developing complex yet user-friendly interfaces, its wide platform base, as well as real-time mission vehicle monitoring and command and control of unmanned vehicles.


Matthew A. Noyes is a student co-op at NASA Johnson Space Center and attends the University of Rochester pursuing a bachelor of science degree in mechanical engineering.


[%= name %]
[%= createDate %]
[%= comment %]
Share this:
Big Data Blogs
Big Data Trends: by David Feinleib