“Pigments of Imagination: An Interactive Virtual Reality Composition”
by Timothy Gmeiner & Eito Murakami.
Abstract: “Pigments of Imagination” is an interactive virtual reality experience that frames the creative process as a narrative relating the synesthetic inner-workings of one’s unique imagination to the universe around us. It is an observation on the beauty, fear, adventure, sadness and loneliness of self-discovery and the artist’s ultimate recognition of process as goal, as told through the story of a small child’s journey to the moon.
This installation is designed to re-imagine the popular music video in a virtual space as a dynamic, emotionally engaging experience by exploring distinctive relationships between spatialization, audiovisual reactivity and interactivity, which allow for a narrative immersion that maintains a structured arc and conclusion but unique experience with each use.
Music: The work is guided by a spatialized fixed media composition that spans genres of ambient-electronic, hip-hop, soul and blues. It was composed and produced with narrative and visual aesthetic in mind through a blend of field-recording, synthetic sounds, sound-design, vocal and acoustic recordings by the author and a host of collaborators. All musical components were then split into individual stems for spatialization and visual mapping. Additional synthetic sounds were created at differing pitches and velocities for real-time cues, providing users with an ability to improvise their own musical elements to the work. The fixed media piece was composed to allow space for such user contribution.
Audiovisual Reactivity & Spatialization: In Pigments of Imagination, a series of parameters for each instrument and sound element were mapped to material, textural and behavior properties of visual assets and systems, allowing for both manual automation and real-time reactivity. For example, a solo flugelhorn was analyzed to gather real-time amplitude, frequency index and frequency value data and then mapped to a series of parameters in an emergent Niagara (particle) system (learn more here).
Spatial audio is delivered by multiple speakers as well as headphones using the Dolby Atmos plugin for Unreal Engine. Key musical elements relative to narrative and form such as voice and bass remain static to the listener, maintaining priority in the mix. Other musical and sound-design elements are spatialized using virtual speakers that maintain fixed coordinates relative to listening location. This allows the experiencer to move their head to listen to the composition from different angles and gain a unique listening experience each time. Additionally, there are dynamic sound sources that coexist with pre-composed music, such as birds and user-generated musical notes; these sounds are attached to their corresponding visual components that follow them around the environment, regardless of listener location and orientation.
Real-Time & Cross-Time Interactivity: In the field of painting, the century-old term pentimento refers to a hidden layer of changes that have been painted over by the artist. In Pigments of Imagination, we apply this practice to the concept of multiplayer engagement; rather than multiple users interacting across space, we allow them to interact across time. Pentimento provides an opportunity for the experiencer to collaborate with the sonic and visual creations of the prior user as they are being created, and build upon them.
This is achieved through interaction with a virtual paintbrush, which can be operated by the current user’s controller to output musical and visual elements that dynamically affect the composition and environment. Output from the previous user’s virtual paintbrush is created in real-time with respect to their visual and sonic location, allowing the current user to collaborate with the previous user’s output to further affect those changes.
The visual design of Pigments of Imagination aims to promote participation by the user. The narrative of this journey begins as a passive experience on a two-dimensional screen within the virtual space, which then smoothly unfolds into three dimensions as a means to reinforce the transition into a continuously evolving environment. Many of the objects that are created by such a progression can be manipulated using the controllers to affect both visual and sonic characteristics of the world. Upon grabbing objects in the environment, haptic sensations are activated in the controller to provide another dimension of interactive freedom to experience the narrative.
Presentation: This experience was presented on May 18, 2023 at UC-San Diego & Qualcomm Institute’s Calit2 Auditorium through an 8-channel system with real-time visual rendering projected to an 18x32 ft screen. The video above offers an introduction to the piece, it’s purpose within it’s field and a deeper explanation of the above technical components, ending with live performance and Q + A session. Patrons were then allowed to enter Pigments of Imagination for individual experiences.
Accolades: In October 2023, “Pigments of Imagination” received an award for best demo presentation at the ACM Symposium on Virtual Reality Software and Technology (VRST) 2023. In 2024 it was a featured experience at the La Jolla Playhouse WOW Festival in San Diego, CA.
Acknowledgements: We would like to thank Gabriel Zalles-Ballivian for his foundational instruction and development on the first iteration of Pigments of Imagination WebXR and subsequent documentation. We would also like to thank Shahrokh Yadegari and Sonic Arts R&D at UC-San Diego for providing guidance, insight and resources toward development. We would also like to thank UC-San Diego Professors King Britt and Steph Richards as well as Zachary Konick and Nick Tolford for their musical contributions to this piece. We would lastly like to thank Keenan Parry and Aamir Asadi for their visual contributions.
Contact: Timothy Gmeiner: tgmeiner@ucsd.edu | Eito Murakami: eitom@stanford.edu