Narcís Parés, Roc Parés - An interaction-driven strategy...

Roc Parés

Good morning everybody. Thank you. Narcís and I are very honoured to be here at this very interesting symposium and I must say particularly in this city in which there is a lot of interest in digital art and technology. Let me say that of course we are not in Finland, we are not in Silicon Valley, we are kind off the beaten track of the hot spots of the digital culture map. But we are working hard on three specific aspects—involving creating art with new technology, teaching and carrying out formal scientific research on this topic as well.

Narcís and I have been collaborating since 1993 on a project called Galeria Virtual (Virtual Gallery) and what we want to communicate to you today is part of this experimental production work and the research behind it as well. We will start running quite quickly through this first half of the presentation, which is only using the adaptation of what we did as a first introductory virtual reality experiment in 1993—when we were working as an independent team at the Science Museum here in Barcelona—, and go on to what we call the Interactive-Immersive Galeria Virtual Demo, in which we try to position our work as artists working with this new interactive audiovisual medium, taking it as far as possible away from the idea or the strategy of mimicry, which we understand as the duplication of human perception of the world.

So what we are going to do now is look at what we adapted from the original experiment, which used a head-mounted display. We immersed the participant into this VRML experience, which you can try yourselves at home on the Web.

Image 1. Interactive-Immersive Demo of Galeria Virtual

What we are trying to do is systematically have this experience organised in a sense in which the user or participant could feel the difference between the way we wanted to approach virtual reality technology as an art medium and what art production and diffusion have traditionally been in the physical world. So we start at this point at which there is a compass on the floor pointing north, as a symbol for the geographical orientation which happens in physical spaces.

Image 2. Beginning of VR experience: compass on the floor pointing north

And if we press this button we see how we lose our geographical orientation. This is only suggesting that in virtual reality artists working on our platform, Galeria Virtual, should be looking for the specific ways in which virtual space is organised, and trying to keep away from the way in which physical space is organised.

Image 3. Loss of geographical orientation

What we can also see is that we are interacting with this virtual environment through a virtual model of a subject which is a position in space which can be moved, and it is also the footsteps of someone who is walking around an urban environment in this first phase. The virtual subject model has a relative dimension which respects objects in this schematic environment, so you are shorter than this stop sign and you are shorter than the buildings, of course. What we can see is that in physical urban space, for instance, we get all these signs, which include the zebra crossing and the stop sign, that make functioning possible in physical world. We were asking questions about what signs would organise functioning, behaviours, and so on, in virtual reality.

The same happens with time and climatic factors. In virtual reality we would like to get rid of the linear sense of time that as human beings we have in the physical world, so we only suggested at this point possible changes in what we understand as the linearity of time and the progressive changes of night and day that we perceive in the physical world.

Image 4. Building with elements of neutral architecture

We walk around this environment, we find this white building which resembles the typical architecture for the communication of art—an art gallery or an art museum. It has all the elements of neutral architecture, an architectural space that has electric lighting, a flat floor, white walls. The goal of all these characteristics of such architecture for the diffusion of art is neutrality. What we thought is, what happens to objects such as this stone sphere or that wooden block, which could be sculptures in space in an exhibition museum in the physical world? So let us look at how in virtual reality artists could not only work with new objects, trying to determine what the specific objects of virtual reality are, but also trying to find out what the specific behaviours of these objects would be, getting away from the idea of being physically lost.

Image 5. Objects in a virtual environment have a specific behaviour.

Then we can see how these objects have been mapped with textures that are digitised textures from known material in the physical world as well. If we press a button the objects lose this reference to, for instance, the wooden block—the wood is a reference to a tree, a tree to a forest, a forest to a mountain, nature, the world, etcetera.

Image 6. Objects lose their reference to the physical world.

So we deconstruct these conditions and we try to find out what may be continuing projects for minimalism and other tendencies in sculpture in the 60s and the 70s, which were already looking for a more precise sense of what a specific object could be, in the terms of Donald Judd, talking about specific and unspecific objects.

Image 7. In virtual reality transit spaces will probably disappear.

We are now in a transit space, which is a glass corridor. In architecture we get these transit spaces because we have to move from one room to another. And in virtual reality in fact we thought that transit spaces were something that would probably disappear because we can move from one area to another, maybe by linking to different environments or whatever.

Image 8. In a virtual environment there is no such distinction between art work and art space.

In this small room we have three monochromes hanging on the walls. We wanted to understand that in a virtual environment there is no such distinction between art work and art space. For instance, the three monochromes hanging on the walls may disappear, their perimeter and their surface being the two limits of the art piece that made them different and distinguishable from the art exhibition space.

Image 9. The monochromes hanging on the walls may disappear, their perimeter and their surface being the two limits of the art piece that made them different and distinguishable from the art exhibition space.

But in virtual reality we want to work with artists who find that continuity in developing concepts of space, objects, and so on.

What we can see now is that if we have been deconstructing the conditions of objects, space and time, everything in fact that is exterior to the virtual subject, we can also work with the suppression of corporeity—one of the properties that gave the spectator a limited experience of this space, limited because he could not walk through walls, the ceiling or the floor of the environment. Now by losing the corporeity of the virtual subject we can have a totally different perception of what is still a schematic model of the virtual environment.

So what we really wanted to do is this, which we did by starting over, saying "Ok, let's forget about this metaphor of physical space, let's try to go and search for the specific properties of virtual environments."

Image 10. Virtual reality, when it abandons mimicry of the physical world, can be a way to experience concepts that are mathematical, such as Cartesian space.

So we have this model of a Cartesian space in which we can see these three vectors in three different colours—RGB—, and with the three last letters of the alphabet determining the three organising vectors of Cartesian space.

Cartesian space, as we are seeing now, is something that as a concept we have experienced only in a limited way by seeing printed diagrams, models or schemes, or maybe on a blackboard when we were learning maths at school. But we have never experienced Cartesian space as human beings. We see that virtual reality, when it abandons mimicry of the physical world, can also become a way to experience concepts that could be mathematical. It is in the spirit of one of the founders of virtual reality, Ivan Sutherland, to give phenomena to the world of numbers in the same way that we have an experience of the physical world.

But we didn't want to end with this idea that the non-limitation that we were looking for ended in this more conceptual world of Cartesian space, because we believe that science and literature, but also cinema, have found their own strategies for creating their own space, different to physical space, and not necessarily Cartesian space.

Image 11. We can have other kinds of hyperbolic space, or non-Euclidian geometry, like fractal geometry.

We can have fractals, spaces, we can have other kinds of hyperbolic space, or non-Euclidian geometry, like fractal geometry, so we also decided to tear apart this residual mental constriction of Cartesian space, and ended up opening our platform, Galeria Virtual, in this virtual void, which is the starting point that we use to invite artists, people who think and work, practitioners and theoreticians, who are trying to work in the specific properties of audiovisual technologies as far removed as possible from the old, clichéd metaphors of buildings and architecture, and as far removed as possible from the metaphor of printed media, especially the book.

So this was the starting point for our research and experimental production work. It was already not a text but a virtual reality experience, and from here we jump to a piece that we made in collaboration with Perry Hoberman—an amazing, New York-based artist. The piece is called "El ball del fanalet" and we are going to use it as a case study for the paper that we are presenting today, describing the possibility of having an interaction-driven strategy in order to develop a virtual reality experience as opposed to the idea of developing virtual reality experiences through a content-based approach.

Narcís is going to present this and after that we have a short video to illustrate it.

Narcís Parés

We have been doing all our production and our research at the Audiovisual Institute of the Universitat Pompeu Fabra. And the aim of this institute is to carry out research and experimental production as well as teaching in the various digital media. We are responsible for exploring the properties of VR as an audiovisual medium, as opposed to a simulation tool. So from there, we have tried to move away from simulation to explore what the specific properties of this medium are. In this research, and in production, we have formalised a new approach to application design for virtual reality experiences.

VR application design is usually guided by a content-driven strategy, which gives priority to the application content and context. That is, the developers have an idea and they focus on that idea to guide other development, including conceptual development, and put those ideas into practice. So the topic defines the context, and the context is absolutely vital for metaphor and interaction elements and interface design.

The basic phases of a content-driven strategy are first to define the application topic or theme, then define the type of application and the type of user. From there several things have to be identified like the virtual objects necessary, the data involved, the processes that will be running, input and output interfaces, object modelling tools, and the application development tools. This strategy can also be described as a "top-down strategy" because we are going from a high level of abstraction to the low level of development at the tool level.

But this content-driven strategy has very evident limitations for people that are experimenting with artistic applications or people that want to work with and study the specific properties of this medium and the interaction qualities and interfaces that can be designed. So from here we define the interaction-driven strategy.

When we develop an application concentrating on how the user is to interact with the application we call it an interaction-driven strategy. We still have no content in mind, we only know what type of interaction we want the user to have. We want certain specific patterns of interaction between the user and the virtual reality experience, so we first analyse interfaces, we analyse the interaction with the elements, the participation, manipulation or contribution of the user, and that results in a spontaneous emergence of the specific content or topic or aroma, if you will, of the application.

The basic phases of this strategy are to start off with the input interfaces, the output interfaces and the type of user that you want to work with. You then define the type of application. And from there you get, if it should be required, the application topic or theme which lets you identify metaphors and all the things that revolve around this. And then you go on to the low level of development, which involves defining the processes, the virtual objects, and so on. So in a way, you could also call this strategy a "bottom-up strategy", although it is not the exact opposite of the content-driven strategy.

This interaction-driven strategy was found experimentally because this piece, "El ball del fanalet" or "Lightpools", by Perry Hoberman and ourselves, was designed in this fashion but not really consciously so. While we were developing it, we started off by defining what we wanted the users to do and only later got all the topics and it was during this process that we became aware of the existence of this new strategy, that we were approaching virtual reality application design in a different manner.

We will now see a short video on the piece, and then we will end by summing up and seeing how this strategy emerged from this piece.

Image 12. "El ball del fanalet" or "Lightpools"

[Video]

"El ball del fanalet" or "Lightpools" is a multi-user experience that uses Virtual Reality (VR) technology. An important component of the work is the exploration of the social possibilities of this medium.

It takes place in a circular arena approximately six meters in diameter, on to which a real-time computer generated image is projected from above.

Image 13. "El ball del fanalet" takes place in a circular arena.

Image 14. A fanalet is a paper lantern typical of a Catalan popular dance.

On entering the arena each user is given one of four fanalets, a paper lantern typical of a Catalan popular dance.

Each fanalet contains a coloured light with a battery pack and a position sensor. The sensor reports its position to a host computer, allowing each fanalet's position to be tracked in three-dimensional space.

Image 16. A sensor allows each fanalet's position to be tracked in three-dimensional space.

The two horizontal dimensions are used to position a coloured circle of light projected on to the floor directly below each fanalet. The third dimension (height) is used to determine the size and brightness of the lightpool, so that its behaviour mimics the effect of a light source emanating from the fanalet: as the fanalet is lowered, the pool becomes smaller and more intense; as it is raised, the pool becomes larger and dimmer. This gives users the impression that the lightpool is projected directly from their fanalet, and gives them an immediate, intuitive sense of how to interact with the work.

Image 17. The two horizontal dimensions are used to position a coloured circle of light projected on to the floor directly below each fanalet.

Each lightpool is a kind of window on to a virtual ground plane, which otherwise remains shrouded in darkness. Small coloured tetrahedron ("proto-objects") are spontaneously and randomly generated throughout the arena, glowing briefly like embers before disappearing.

Each proto is matched in colour to one of the four fanalets.

If a user manages to illuminate the proto during its brief life with the appropriate lightpool at a sufficient intensity, the proto grows and metamorphoses into an articulated object. In effect, the object feeds on light.

Image 18. The proto-object grows and metamorphoses into an articulated object.

The objects range variously from mechanical to biomorphic, abstract to ornamented. When the object has reached sufficient size, it remains stable, and is thereafter under the control of the user. At this point, if the user fails to decrease the intensity of the light, by raising their fanalet, the object grows until it bursts, scattering a new crop of coloured protos on to the floor. These protos can then be nurtured by other users, and the cycle continues.

Image 19. When the object has reached sufficient size, it remains stable, and is thereafter under the control of the user.

Once an object has grown and stabilized, it becomes the user's "partner". Slow movements of the fanalet are interpreted as "leading movements", which the object follows as it dances around the lightpool.

Any sufficiently rapid sequence of movement of the fanalet is interpreted as a "training movement"; the object follows this movement, and then repeats it continuously on its own, until it is taught a different movement.

By alternating rapid and slow movements of the fanalet, users can teach, dance and interact with their object. Although the projected image is two-dimensional, the objects themselves have been modelled in three dimensions, so that upward and downward motions of the fanalet affect the object's orientation and scale. Each object is made up of three distinct parts, which follow the motion of the fanalet at slightly different rates; this gives the objects a somewhat fluid appearance; certain details are only revealed when the object is in motion, a bit like a peacock showing off its feathers.

Image 21. The objects have been modelled in three dimensions, so that upward and downward motions of the fanalet affect the object's orientation and scale.

From these interactions emerges a complex, ever-changing dance of participants and virtual objects. New users can enter at any time, as long as there is an available fanalet, and users can stay as long as they like. Additional spectators can watch comfortably from behind the circular railing that encloses the arena. Several users can share a single fanalet, passing it back and forth as they wish. The atmosphere is one of a casual but concentrated chaos, as users observe and interact with their objects and each other.

Image 21. The atmosphere is one of a casual but concentrated chaos, as users observe and interact with their objects and each other.

Each event in the piece has an audio component as well: a delicate glassy tinkle as protos appear, the whoosh of a pneumatic pump as objects grow, whipping air currents as objects are trained, and so on.

A second independent stereo audio track, consisting of a variety of ballroom dance music, processed with huge amounts of reverb, plays continuously and softly in the background, as though it is being heard from a great distance.

At any point, after two or more users have grown and trained stable objects, they can bring their fanalets together in a ritual gesture that indicates that they want their objects to join together in a choreographed object-dance. After a brief fanfare and a shower of light, the lightpools and objects leave the user behind and perform a group dance to a driving drumbeat, using the movements that they have been taught as they follow a choreographed path. The lightpools then leave the arena, and users can begin the cycle again.

[End of video]

The fact that we started off from the "El ball del fanalet" metaphor might mislead the observer. We in fact started by defining that we wanted a projection on to the floor from the ceiling. We defined that we wanted it to be a multi-user experience. We defined that we wanted the users to explore the floor, the environment projected on to the floor; somehow we still didn't know it would be with fanalets. We knew we wanted the multi-user experience to be a social experience, so users would not only meet through the virtual environment but also directly face to face, which is quite unusual in multi-user experiences. And from there we started thinking that it would be a good idea to explore the environment through lightpools. And from there we thought that the metaphor of light and the social aspects of the installation were appropriate to incorporate the metaphor of "El ball del fanalet" as a popular Catalan dance.

Roc Parés

Well, to conclude, our interaction-driven strategy was found experimentally. Content-driven and interaction-driven strategies work under different frames of reference. One gives a complete twist to the other. An interaction-driven strategy provides more freedom in interaction and interface design. On the other hand, a content-driven strategy is less flexible, but is also more controllable and rigorous in the definition of requirements because you have a specific content from which you want to define a requirement.

And we should also point out that these are not the only two possible strategies and that many others can be specified, but for the time being we have found and formalised these two.

And as we said before, if you want more information, more details on this, we have an article explaining all about this strategy, the formalisation and technical aspects, and on the Galeria Virtual and "El ball del fanalet" websites you'll find more information on this.

Thank you very much.