Passing Through the Heart, 2020
Live-Stream Performance & Workshop Series
ISEA International
Motion Capture, Real-Time Rendering
Pass Again Through The Heart: Gesture, Memory, and Food was a live-stream performance and workshop series, presented at the 2020 ISEA Symposium, an international symposium on electronic arts. This project was a series of workshops that utilized technologies like photogrammetry, non-optical motion capturing, videography, and audio recording, to support oral testimonies, and transmedia storytelling centring around food as a cultural practice. I was contracted by the Public Visualization Lab to develop a multi-channel live-streaming environment that would be used for the online performance and workshop series.
The completed set included a real-time rendered 3D set that contains an environment with an actor wearing a motion capture suit, and several live-action video feeds transmitting to a zoom channel. A management system created in Touchdesigner contained the digital set, with multiple outputting cameras. Another Touchdesigner instance managed the live-action camera feeds. A diagram of the setup and photos can be found below.
Role: Touchdesigner & UnrealEngine Developer
Organization: Public Visualization Lab (OCADu)
Technology: Unreal Engine 4.25, OBS, Touchdesigner, Axis Neuron Motion Capture Suit
Passing Through The Heart: Episodes 1-3
The set was used for the ISEA workshop as well as to capture video and motion data for the subsequent series of workshops titled Passing Through The Heart: Episodes 1-3 which was presented at the Squeaky Wheel Film & Media Art Center, Western NY. For This series, I re-designed the digital set to use Unreal Engine 4.25 to take advantage of the program’s advanced rendering and lighting features. The raw video was further processed and edited by Professor Immony Men, links to the completed series can be found below.
PASS AGAIN THROUGH THE HEART: GESTURE, MEMORY, AND FOOD {Live/Realtime}
w/ Immony Mén and Patricio Dávila, October 17, 2020
Passing Through the Heart: Episode 1
w/ Immony Mén and Patricio Dávila, November 13, 2020
Passing Through the Heart: Episode 2
w/ Kalpana Subramanian, December 11, 2020
Passing Through the Heart: Episode 3
w/ Cheng Yang “Bryan” Lee, February 26, 2021
Installation Photos & Diagram
Compatibility Issues with TouchDesigner, FBX models and Axis Neuron
An Axis Neuron Motion Capture Suit was used to capture the movements of the workshop facilitator Professor Immony Men. To create the real-time motion character rendering, it was necessary to develop a technique for connecting the motion capture suit (Axis Neuron) to Touchdesigner as no prior documentation exists. There were issues with the model’s skeleton and rig, the model would become distorted or broken when parented with the motion data. I created a solution using Blender and Mixamo, the documentation of the solution is recorded in a Youtube video found below.
