> Encounter (Resonances)
Hayley Hung (Univ. Amsterdam) & Christian Jacquemin (LIMSI & Univ. Paris-Sud)
On-line video (format MPEG2) Encounter_ACMM2010.mpg
In this work, we attempt to reproduce one of Mark Rothko’s Seagram murals. Using computer graphics, optical effects and reproductions of his work, Encounter (Resonance) questions whether a reproduction can be more than it seems at face value. The work provides an analogue for Rothko’s mixed media painting technique by using reproductions of Rothko’s “Red on Maroon” found on the internet. While each of these copies may instantly remind us of the original work, the grapical rendering of Encounter (Resonances) combines them at three levels of representation (global shape, micro and macro structure), and offers a multi-faceted view on one of the most famous artist’s work.
In the 1950s, abstract expressionism began and artists such as Mark Rothko and Barnett Newman began to question the divide between the onlooker and the canvas. They believed that their artworks were themselves spiritual entities, rather than metaphors or similes of them. Through this philosophy the relationship between the artwork and onlooker was transformed and the meaning of the paintings themselves became a dialogue between the individual and the work. For Rothko, his last works were painted in very thin layers of paint which had an iridescent quality only when viewed in darkened conditions where observers could open their senses to the possible interactions between themselves and the paintings.
“Since I am involved with the human element, I want to create a state of intimacy -an immediate transaction. Large pictures take you into them. Scale is of tremendous importance to me -human scale.” 
More recently, the combination of art and digital media has given an important role to the augmentation of the human body. Intrusive augmentations such as exoskeletons (Stelarc) or an injected computer chip (Eduardo Kac)  could only be performed on the body of the artists themselves. When interactions involves members of the public, body augmentation is generally achieved through virtual renderings (sounds and images) and rely on video capture and/or shadow projection. One of the important issues in such artworks is finding the connection between the human body and its virtual counterpart. Golan Levin in Messa di Voce  used the projected shadow of the singer and augmented her image on the screen with a visualization of the voice. In Bubbles (Wolfgang M¨unch and Kiyoshi Furukawa)1 the shadow of the user interacts with virtual bubbles. Even though such connections may give a sense of presence and engage the onlookers in playful interactions, we believe they lack the establishment of a strong connection between the physicality of the human body and the artwork. Also, the consciousness of being physically part of the artwork itself is, to our knowledge, is not always addressed, established or explored.
Encounter (Resonances) is an interactive installation that questions the authenticity of a reproduction by creating an interactive experience between a viewer and the artwork. In this piece, one of Mark Rothko’s Seagram murals has been reproduced. The work deconstructs Rothko’s painting and recreates it based on various reprodutions of the painting found on the internet. These reproductions form the basis of the iconicity of Rothko’s painting. By using these coarse approximations of his work, we create an installation piece which questions authenticity and the relationship between the onlooker and an artwork. The interactions in this work take two forms; In the first stage, a person approaching the work realises that when they come close enough, the painting begins to change relative to them. Specular lighting moves relative to the viewer and also parrallax effects are used to immerse the viewer which are composed of a series of layered structures.
If the viewer is willing to engage with the work for a prolonged period of time, a second stage of interaction occurs where the work starts to slowly reveal its layers. In these layers are the imperfections of the digital reproduction of the painting, which cause scintillating optical effects which can only be perceived if the viewer is completely engaged with the work. They are distorted interpretations of the thin layers of paint that Rothko used to create the subtle effects in his paintings. Encounter (Resonances) has been created to encourage prolonged engagement through discovery in interactive art. It promotes patience and a willingness to perceive beyond the surface or initial appearance and questions fidelity and authenticity when things could be taken at face value. The complexity of the visual effects occur at various scales and involve subtle and diffuse changes in the painting light, color, and shape. Because of the subtlety of this installation, each viewer must create her own personal dialogue with the work if she wishes to understand it fully.
In questioning the authenticity of reproduced artworks, a question is raised about how the essence of an artwork is lost when it is reproduced and how this damage be repaired to regain some of the lost authenticity of the original artwork. For artists such as Rothko, much of his success for the abstract works which is he is most famous for, captured the imagination of critics through his use of colour; something which he resented deeply. However, while colour is an important part of Rothko’s work, the often iconic representation of his paintings, reproduced in post cards and prints, can only show one specific surface view on his work. While reproductions allow an artwork to become accessible to more people, it is often easy to overlook what is lost; colour, scale, texture, subtle light scattering due to the painting layers, and perhaps even interaction are some of the aspects of an artwork that appeal to our senses. In our artwork, we purposefully use the colours of Rothko’s ‘Red on Maroon’ (1959) (Figure 1)which have been taken from reproductions on the internet. While these are not the original colours of the painting we try to recreate an artwork that can still try to capture the original goal of Rothko’s work by encouraging engagement with the artwork. While colour was important to him, texture was also extremely important. The layering technique that he used for his paintings encouraged viewers to look carefully at the painting. The layering technique in itself causes the perceptions of the colours on the canvas to change as the viewer moved, as light reflected off different paint surfaces.
3. TECHNICAL DESCRIPTION
The installation uses real-time video processing and simultaneous computer graphics generation. There are two stages to the set-up involving video analysis and image synthesis which renders a graphical representation of the painting. More details about each part are explained in this section.
3.1 Graphical Rendering
The graphic rendering of this installation relies on a very standard architecture of a pile of layers and attached masks (see Figure 2). Such an architecure is used by most image processing tools such as Photoshop or Gimp. It is well-suited for combining and blending several images through the superimposition of semi-transparent layers. Each layer has two masks: a partial mask for non-uniform transparency blending, and a dotted mask for moir´e effect through layer rotation. In addition, the top layer has a normal map for specular lighting. It is therefore necessary that this layer should not be fully transparent.
The graphical rendering is implemented in the 3D engine Virtual Choreographer, through a single quad with as many textures as layers, masks, and normal map. Fragment shaders are used to compute the mask-based transparency, pixel holes, and the specular lighting of the top layer. Vertex shaders are used for GPU computation of texture rotations.
3.1.2 Multiscale and Multilayer Moiré Effect
The art installation was intended to be accessed through three levels of details to the onlookers that would correspond to a progressive discovery of the artwork. Each level offers a different conceptual and sensorial approach which echoes the style of Rothko to light and color through sophisticated multilayer opaque or translucent painting.
The first and general level is a global view of the painting that can change color with time by combining the contributions of all the samples of the painting found on the Internet. For this purpose, the views were all scaled (without interpolation) to the same dimension; approximately the screen size of the painting. This normalization process results in images with different levels of resolution; the smallest of them is quite pixellized when scaled up to the proper size. Through transparency effects, these layers can be blended. They offer the viewer a constantly changing representation of the painting by varying continuously and independently the levels of transparency of each of the layers. So that the transparency does not blend uniformly across all the painting reproductions, it is combined with a set of black and white masks that hide or reveal parts of the image when blending them. Each mask relies on the image lightness (the dark zones are more transparent than the bright ones) and only covers a subpart of the painting. Figure 3 shows the seven masks applied to the painting layers in color so that they can be easily differentiated. These shape masks have been derived from the chemical analysis of the artwork, which was obtained from the Tate Modern in London. The chemical analysis allowed us to extract an authentic separation of some of the materials used when Rothko was applying thin layers of paint to his canvas.
The second level concerns the macro structure of the artwork that can be highlighted through specular (reflective) lighting. For a fixed light source (such as the lighting in an art gallery), specular lighting depends on the position of the onlooker with respect to the painting. It is revealed by moving around the painting and engages the viewer to adopt different angles of view to perceive it more extensively. To be efficient, this effect supposes that the position of the onlooker is tracked and used to parameterize the specular lighting according to her/his position in front of the painting. The specular lighting is controlled by the relative position of the viewer and light direction with respect to the surface normal. By modifying the normal, a macro-relief can be simulated as shown in Figure 4.
The third and finer level of detail concerns the microstructure of the work, typically the pixel level since there is no finer uniform level of detail in such a video-projector based installation. The purpose of the rendering of this small scale level is to allow viewers to perceive moiré effects due to the relative positions of pixel-wide holes on each layer. These effects should evoke further the interactive effect of the subtle transparencies of Rothko’s paintings. They are produced by independently rotating very finely each layer. Thus, moir´e effects result from the variation of pixel colors depending on whether the top layer(s) have a hole located on this pixel or not. To enhance these effects, the rendering of the painting textures and masks is not interpolated so that the pixellized surfaces are not blurred through interpolation averaging. A sample dotted mask used to make some of the transparency according to the layers. The layer pixel transparent is shown in Figure 5. Through this rendering, complex patterns can emerge that can trigger the imagination of the viewers .
The observer stands in front of the screen where she will be captured by a camera above the screen. The video capture and the associated image analysis software is used estimate where she is in relation to the painting, and also how long she has been standing in front of it. Both these aspects of the behaviour of the onlooker is captured using standard background subtraction and silhouette modelling techniques. In its current form, the system works best if only one person is allowed to interact with the artwork at a time due to the nature of the design of the piece.
The output of the video analysis is sent to the image synthesis application through UDP network messages encoded in a binary and compact format (Open Sound Control). The messages are used for the following controls on the multilayer image/mask compositing:
• the motion of the viewer is used to translate the layers according to their depth. It generates small parallax effects to give her a sense of depth and layering,
• the viewer’s position is also used to modify the specular light seen on the painting (the reflection of the virtual light source on the painting surface layers),
• last, the absence of strong motion triggers the subtle relative rotations of the layers to generate microscopic color changes (noticeable at the pixel level) through moir effects.
Contrary to many digital artworks in which interaction generates strong changes in the work behavior, and wish to engage in a communication with the onlookers, our soft and subtle approach to interaction could be called environmental. By observing how the user enters the installation environment, she is progressively immersed into fine variations of the visual rendering.
Encounter (Resonances) offers an interactive experience of discovery where the face value of a reproduction of an artwork is questioned. Using graphical rendering and computer vision techniques to realise this installation the work tries to emulate an authentic experience with one of Rothko’s paintings where enjoyment of the painting can be deepened only with prolonged engagement and with a willingness to perceive beyond the initial impression of the work.
 J. E. Breslin. Mark Rothko: A biography, chapter 13. University of Chicago Press, 1993.
 S. Dixon. Digital Performance: A History of New Media in Theater, Dance, Performance Art, and Installation. The MIT Press, Cambridge, Ma, 2007.
 G. Levin and Z. Lieberman. In-situ speech visualization in real-time interactive installation and performance. In Proceedings of NPAR’04, pages 7–14, 2004.
 T. M. Reagan. The study of patterns is profound. Leonardo, 40(3), 2007.