Mocap Experiment: An Agile Way to Explore Virtual Dance
我從好奇甚麼是「虛擬身體的在場」（the presence of virtual bodies）開始。對舞蹈x技術領域的濃厚興趣促使我於2020年2⽉參與了由城市當代舞蹈團組織的網路駐留計劃，與國際藝術家在線合作。隨後我編排與跳《渡》探究真實與虛擬的邊界，運用頭戴式顯示設備來帶觀眾及舞伴進入想像的世界，曾以PCVR版本及劇場形式呈現。2020年12月我加入Eugenia Kim組織的Leonardo21藝術研究駐地，發起一系列實踐包括mocap-ing、設計虛擬現實世界、與跨學科學者進行討論等。
在技術的硬件越來越先進、軟件及內容⾼速發展的環境，影響表演藝術的範式轉移。例如第一視角的可及性轉變了本體感覺。我意識到研究虛擬舞蹈，需要花精力去大膽探索甚麼是身體。首先得先了解動捕，此技術可記錄物體或人的運動的過程，被應用在醫學、遊戲和教育等領域。動捕包括光學系統、使⽤附著在舞者⾝體上的標記或無標記的深度動捕相機等。我使用的是⼀套無線傳感器套裝Rokoko Smartsuit Pro。
動捕延伸的活動：2021年7至9⽉我被聘為 ROXY - TanzLabor 遠程數字舞蹈創客，在⼀個劇場的裝置中進⾏AR舞蹈表演。21年9⽉我在柏林數字⼈⽂藝術的研討會分享經驗，視頻稍後會公開。
2021年4⽉，我以客座藝術家身份與瑞士XR先鋒藝術家Gilles Jobin及其團隊合作，擴展想像力的邊界。我參與的La Comédie Virtuelle是多用戶社交VR體驗，前期與三位舞蹈藝術家合作排練。公眾可通過虛擬現實設備（Oculus和HTC Vive）或電腦參與。他／她們會變成不同顏⾊的火柴棍，與舞者和其他虛擬化⾝相遇，任意走動並聽到不同區域裡的音樂。
Mocap Experiment: An Agile Way to Explore Virtual Dance
Original text: Zelia ZZ Tan
Translator: Penelope Zhou
Keywords: motion capture, body, disembodiment, experiments, senses, virtual reality, virtual dance, real-time performance
Motivation Behind the Experiment
Remote collaboration／Photo provided by Zelia ZZ Tan
My motion capture (mocap) experiment is designed to test kinaesthetic awareness and proprioception in both physical and virtual realities. In practice, the invisible dimension of the environment becomes perceivable. Through the choreography and performance of virtual dance, this mocap experiment sets out to explore how the interaction between the body and technologies can help creators and audiences form a perception of reality.
My exploration of the subject started with the presence of virtual bodies. I have always been interested in the possibilities of the cross-over between dance and technologies, and it was this curiosity that inspired me to take part in the City Contemporary Dance Company’s Online Artist in Residence Programme in February 2020, through which I collaborated with a number of international artists using the power of the internet. Later, I choreographed and performed my work trans-, a contemplation on the boundaries between physical and virtual realities. The piece, which invited the audience and dance partners into an imaginary realm using VR headsets, was presented both as a PCVR experience and a theatre performance. In December 2020 I joined the Leonardo21 artistic research residency organized by Eugenia Kim and engaged in a series of mocap projects, virtual reality designs and interdisciplinary discussions with professionals from multidisciplinary backgrounds.
Today, cutting-edge technologies—advancing at an unprecedented speed both in hardware and software—are creating a paradigm shift in the nature of the performing arts. For example, the accessibility of the first perspective transforms proprioception. This made me realize that in order to explore virtual dance, I must first understand what constitutes a “body”. For that, I enlisted the help of mocap, a technology that records the movement of objects or people and has been employed in a wide range of applications in areas such as medicine, gaming and education. The gear involved includes optical systems, various types of markers and mocap cameras, among others. I opted for the Rokoko Smartsuit Pro, a mocap suit with built-in wireless sensors.
The main goal of this experiment is to search for the fundamental relationship between the body and the act of performing. I revised and improved my creative concept based on my experience as a dancer and external feedback. It took some time for me to turn my mental stress and the challenges I faced into practicable ideas, but this process was instrumental in helping me digest all the new knowledge and experience.
The Mocap Process
When I danced wearing the mocap suit, I imagined that the sensors were attached to my joints and bones and were moving with my muscles and nervous system. The suit has 19 sensors and the movement data that they collect is processed by the computer to construct a brand new bone structure. This avatar allows me to present the dances of my imagination in another space, and to break through the limitations of the physical body, turning volumetric movement trajectories into digital data without having to rely on 2D imagery. As I practiced more, I developed a better awareness of how my entire body was being tracked and was therefore able to dance in a way that could translate into more meaningful behaviour in the virtual environment.
Mocap data application／Photo provided by Zelia ZZ Tan
Meanwhile, various glitches involving disembodiment helped me gain a better understanding of the features of the equipment—for example, when my knee and my head moved in different directions simultaneously, my virtual avatar started sliding. In order to have maximum control of my avatar’s movement rather than having to spend a lot of time on data clean-up, I needed to have a clear understanding of how to perform my dance so it could be accurately rendered digitally. I also tried reversing this process, by first improvising various tasks, then combining and inputting the mocap data into the game engine, and interpreting the actions of the avatar in the following steps. What I discovered was that the features of the mocap equipment do not have a direct impact on my creative intentions, because I used a less technical approach and it did not provide me with profound sensory feedback. Although the mocap suit as an object can be choreographed and used for different purposes, to explore the strategies provided by performance partners and communicate with technical artists, in fact many unknowns remain to be discovered.
Related mocap activities: From July to September 2021 I was hired by ROXY – TanzLabor as a Distant Digital Dance Maker to perform AR dance in a theatre setting. In September 2021, I shared my experience in Digital Research at the Humanities and Arts conference in Berlin, of which the video clip will be made public later.
Real-time Virtual Dance Performances
In April 2021, I started a collaboration with Swiss artist Gilles Jobin—known for his cutting-edge work in XR (extended reality)—and his team as a guest artist. The project, La Comédie Virtuelle, was designed to be a multi-user social VR experience whose goal was to expand the boundaries of our imagination. Prior to my involvement, three other dancers had participated in it. The public could access the experience using VR headsets（Oculus and HTC Vive）or their computers. As they enter the virtual reality, viewers are transformed into matchsticks of varying colours and have encounters with the dancer and other avatars; they can walk around freely and hear the music in different areas.
Real-time virtual dance involves a combination of various equipment. Every performer needs to discuss with their technical team how they want their digital body to move or transform. Through a method that I describe as “invisible choreography”, I was able to build trust in this interdisciplinary collaboration and to get a sense of what this hybrid reality experience was like for different people.
In the work, my avatar danced outside the oldest theatre in Geneva, Switzerland, while I was in fact in my own home, performing and generating mocap data in real-time, the footage of which was played alongside the virtual performance to provide the audience with an understanding of how the dance came to be. I set up a series of screens in my home and made sure that my mocap systems were calibrated correctly, so that the viewers could see me. Through observing real-time data and listening to my dance partners, I was able to reduce the discrepancy of movement between the physical and virtual realities, thus achieving more vivid telepathy.
During the performance, when avatars came in close proximity with one another, sometimes they passed through each other’s bodies, or even disappeared from the virtual space altogether. What is interesting is that "calibration", as an option for fade-in or fade-out of the virtual environment, is the choreographed interaction with the technical partner. Namely, the displacement of dynamic data in the game engine is part of the authentic experience of a collaborative virtual dance.
Via my observations on the changes of the virtual body when performing the invisible choreography, I was able to further explore the relationship between myself and the world around me. With new materials and technological agency, digital embodiment can enable participants to take risks, experience a different life, exchange identities, and interact with the virtual environment, giving feedback to the virtual environment, and imagining life with many dimensions.
With the potential and pitfalls of technology continuously altering every facet of life, we are working hard to address the myriad challenges posed by technological advancement, both individually and collectively. After obtaining a deeper understanding of the first principles, the methods and my own interest in this area, I feel empowered to look further into the questions of “What is reality?”, “What is the body?” and “What is dance?”. I look forward to seeing more artists turning their creative vision and research into multimedia, interdisciplinary artwork that is accessible to the public, and sharing their experiences with local and international communities.
Zelia ZZ Tan
A young Asian dance creator crossing over to film and technology. Focused on the purpose, form and method of the body, to explore the truth of art and the world. A professional dancer at the City Contemporary Dance Company. Among the five short films she has directed, Over Here? and Moon Paradox I have been presented at 15 international dance film festivals.