Draw Within Water
Collaborative creation in augmented reality
The feeling of painting your first brushstroke through virtual space is like being awake within a dream.
Draw Within Water is an augmented reality installation at the HoloCenter on Governors Island that we created to enable visitors to engage with virtual immersion. Sky Rolnick developed the application that runs Draw Within Water in Unity. The scene features music by Keith Patchel and artworks by Spacer Arts, Lydia Powell Jessup, Alexandra Patz and Ioana Pioaru.
In this multiple-participant experience you can draw in virtual space and further populate the underwater scene from the library of sea creatures created by the artists. We launched the project in September, 2019 and it continues evolve with new creatures, animation and refinement of the interaction.
For many visitors to the HoloCenter, Draw Within Water is their first experience with immersive augmented reality.
Virtual experiences and holographic imaging have conceptual overlaps and are converging in the growing field of holographic media. There are nuances to each technology that enable distinct modes of visual experience. The HoloCenter encourages artists to explore a range of technologies and develop their own visual aesthetic.
The Center for the Holographic Arts — HoloCenter was established to enable artists to make holograms. Actual holograms. Holograms are physical structures that shape light and are mostly recorded by lasers. Holography and digital imaging are converging — in jumps — and the confluence has enabled artists to print holograms of their digital creations as well as the emergence of a new generation of volumetric video displays.
For marketing, companies are keen to call almost any technology that creates a visual illusion ‘a hologram’ completing ignoring what a hologram actually is. The problem with this misuse of the term is that we lose insight into some really exciting technologies — such a light field imaging and waveguide displays.
The HoloCenter supports artists to work with a wide rage of these ‘holographic’ media, often connecting artists with experimental materials and displays. The artwork KLESHAS by Ikuo Nakamura was developed in collaboration with Looking Glass. This collaboration enabled Nakamura to create a volumetric video that is presented on a Looking Glass display at the HoloCenter.
There are no holograms in Draw Within Water but the illusion is enchanting. To create the ‘augmented reality’ experience we are using Lenovo Mirage headsets which have a visor that reflects a stereo — left and right eye — video feed. The reflected video ‘augments’ what you see in front of you. Augmented reality — AR — in its various manifestations is sometimes described as mixed reality or XR.
We created an underwater scene as a contribution to the growing awareness of our effect on the oceans and responsibility for our actions. This is a theme that runs through many of the public programs on Governors Island. We also chose an underwater scene because putting on the headset is analogous to donning a scuba driving or snorkeling mask to enable the experience of underwater environments. We chose to present this experience using augmented reality headsets so that participants could still see each other and their physical surroundings. The feeling of a collective experience between the participants is palpable.
To enable all ages to creatively engage we wanted a simple design. There is a single control panel in the scene where participants can select brushes, colors and creatures. It was important to us that everything could be done with a single controller as people felt more comfortable having a free hand and younger participants often use their spare hand to support the weight of the headset. To accommodate our youngest visitors we added a submarine with an augmented viewing periscope.
Draw Within Water was also developed as a continuation of Paula Dawson’s artistic research into drawing holograms. Working with the HoloShop group at the University of New South Wales, Sydney, Australia and the Media Lab at MIT, Dawson (now retired) created spatial drawings with the PHANToM haptic pen and digitally printed these as holograms.
There is now a number of much more affordable options for drawing in virtual space. Tilt Brush, one of the most ubiquitous drawing applications, was used by Ioana Pioaru to create her Spectral Figures series of digital holograms.
Four holograms depicting the artist’s influences along with a self portrait are part of the exhibition ‘Holographic Embodiment’ at the HoloCenter. Pioaru describes her process as sculptural drawing. The figures are sublime in their expression and visual fidelity. Digitally printed by Geola the holograms are high resolution surface etchings. The technique enables the images to be seen from both sides, adding to the ethereal quality of the artwork.
Across the hallway from the Pioaru holograms is the installation Draw Within Water. Glowing jellyfish, made from up-cycled plastic bottles and hand-made kelp hang in the room that is lit with rippling lights, and a myriad of LEDs. Put on a headset and another world of coral and sea creatures is overlaid.
Draw Within Water grew out of two research projects. The first project looked at how to bring mixed reality experiences into museum environments in partnership with Kathleen Ruiz, Co-Founder of Games and Simulation Arts and Sciences at Rensselaer Polytechnic Institute. While discussing with artists and curators how to create meaningful virtual experiences of art we kept coming back to the need for artists to be able to experiment with virtual creation tools and for visitors feel comfortable interacting while others looked on. A number of methods for expanding individual experience were designed.
Our second research project was to design a simple technology solution to enable artists and the public to collaboratively draw in a virtual environment for various outcomes. We wanted to create a drawing experience that was simpler than Tilt Brush — where people take a while to explore the multi-faceted menu of brushes and effects. In high traffic galleries and when working with school groups we need participants to very quickly grasp all the drawing tools. We also looked at mobile augmented reality such as Just A Line where you can draw by moving your phone through space. But to enable both an immersive experience and for the drawings to viewed later with a cellphone it became clear that we needed to develop something new.
What began as consultations with industry experts Alexandre Girardeau(highway101etc), Sky Rolnick(Jump into the Light) and Alon Grinsphoon(echoAR) quickly turned from talking to doing.
Sky Rolnick who co-cofounded Jump into the Light — the first virtual reality cinema and play lab in New York — was already thinking about group augmented reality experiences and built the Unity application behind Draw Within Water.
EchoAR’s content management system—CMS — offered an elegant platform for taking the sea-creatures or other drawings and attaching them to a QR code so they can be viewed with a cellphone. The creature appears to hover above the QR code and you can use your phone to look around it. The creature that is attached to the postcard changes every week or so. We also print stickers onsite so that any creature can be taken with you.
Although we wanted to develop an experience for augmented reality headsets the cost of multiple HoloLens or MagicLeaps (which are thousands of dollars each) was outside our budget. Sky Rolnick came up with the idea of attaching a Vive trackers onto a Lenovo Mirage headset and each one has a Google Pixel 2XL phone inside. Most of the technology that we bought was surplus or second hand. Jump Into the Light lent some equipment and Vive sent us a tracker when they were first released. The tracker from Vive sparked a whole range of ideas about creating with the movement of bodies through space.
The HoloCenter often has the wow meter on ‘high’, and this installation is no exception. Most people are comfortable going into the virtual scene, but describe the strange feeling of taking the headset off. After being surrounded by sea-creatures and drawings and navigating the space around them the room felt vacuous. It is was more empty than when they walked in.
The whole exhibition ‘Holographic Embodiment’ explores relationships between the physical and virtual aspects of identity. Many of the works present a coexistence of psychological or spiritual dimensions. In the galleries we encourage people to move around the holograms so as to ‘play’ them. Each hologram suggests a choreography, a specific set of movements, that unfold the imagery.
Holograms such as Dan Schweitzer’s Tunnel (1979) invite the viewer to imagine being immersed in a virtual world; Draw Within Water allows people to create within a virtual world. By exhibiting these works together my hope is that people explore the imaginative dimensions of visual media.
Drawing in virtual space opens up creative practice on a fundamental level. While computer generated imagery — CGI — has enabled artists to create other worlds, the creation and experience is often removed from our embodied experiences. In contrast, sculptural drawing has a physical immediacy. To watch an artist draw is a kind of dance and when viewing the brushstrokes you sense the arcs of a body in motion. The aim of Draw Within Water is to inspire people to create and shape the world around them.
Draw Within Water is developed by Sky Rolnick with Martina Mrongovius and echoAR. Music composed by Keith Patchel. Featured artwork by Spacer Arts, Lydia Powell Jessup, Alexandra Patz, Ioana Pioaru and you!
Draw Within Water is installed at the HoloCenter on Governors Island until October 27, 2019. Free and open to the public every weekend 11am-5pm — check the HoloCenter website for further installations
The research projects were made possible with funds from the NYSCA EMF in Partnership with Wave Farm: Media Arts Assistance Fund, with the support of Governor Andrew Cuomo and the New York State Legislature. Visit the NYS Media Arts Map to find more places that combine technology with creative practice.
