• Etienne Armand Amato

Body and interactivity : Interface, Narrativity and Gestuality

2003 / Information et Communication / Article

This article is been published in the bilingual review Anomalie whose reference is

Amato Etienne Armand, Weissberg Jean-Louis. « Body meets interactivity : interface, narrativity, gesture. », in Interfaces, Anomalie digital_arts, n°3, 2003, under the direction of de Aktypi Madeleine, Lotz Susanna and Quinz Emanuele, p.240-245., édition HYX, ISBN 2-9518811-0-X
Thanks to Madeleine Aktypi for this great work of translation

Credits : E2A

Etienne Armand Amato : Jean-Louis Weissberg, you have coined the term “spectactor”, which merges together spectator and actor [1] (as in the one-who-acts-on-something) up to a point where they become hard to set apart as different figures. You are thus trying to qualify the totally new posture set down by interactive media, whose main characteristic is that they bring together those two traditionally antithetical situations.
However, while attempting to exceed that, the term spectactor carries one back to the traditional distinction according to which the spectator is a passive being while the actor is crowned with an all-positive aura.
And yet, concerning that first point, the classical spectatorial position has been redeemed by studies on reception. When one watches a spectacle, one is implicated in a real co-operation and this collaborative dimension relegates all the alleged spectator’s apathy to simple prejudice.
As far as the actor aspect of the interactive relationship is concerned, one could easily question the nature of the activities at stake. In some cases, those activities look like passive ones merely sustained by an infra-conscious mental investment. Automatic gestures - I am thinking of vehicle piloting or mechanical firing and other reflex movements - rather put the participant in a somnambulistic state than in a conscious action one.
On the other hand, the much talked-about human-computer “dialogue” is based on an action-reaction pattern, which allows testing and evaluating the interactive interface thanks to the responses it sends back. In that context, the spectactor simply asks the questions the machine can allow for. Fallen from the status of the one who acts, he resembles more to a clumsy operator or a mere instruction executor, who only imagines the freedom he enjoys. All those remarks taken into account, could you clarify your present idea of the spectactor ?

Jean-Louis Weissberg : The “Action on image : for the elaboration of a critical vocabulary” seminar [2] has helped me, through such pertinent criticism, realize what I used to envisage by this risky term. I should first insist on its limited field of application. I think that the term cannot cover in a valid way the whole range of what is called “interactive” activity. By the way, I would like to specify that I believe we should differentiate between interactivity and interaction ; the later being a much larger notion than the former which should specifically concern informatics exchanges. Can we for instance argue that someone who uses a search engine on the Internet is a spectactor ? The very idea of spectactor finds its greatest pertinence in a quite limited area within interactive activities, that of the narrative spectacle, of fiction in the largest sense, including some video games. In fact, it is the film spectator posture, and in a lesser extent the theater one, that made me coin the neologism spectactor.

E.E.A. : Does that mean to say that narrative spectacles - after having taken hold of analogical recording and representation technologies such as film, radio and television - have also naturally invested the computational abilities so as to shape a spectactor posture which then simply arises as the result of such a meeting ? If that is the case, then the spectactor is just a by-product of the growing process of spectacularization of the technical innovations which the progress and leisure ideologies recycle in an attempt to provide the individual with an illusion of mastery compensating for loss in control concerning the self, the world, and most of all the political.

J.L.W. : I would rather insist on the “line of descent” which distributes the principal expressive fictional vectors : theater, literature, photography, film, television, video, hyper-fiction. I mean that the question of the receiver’s place is always a techno-cultural matter. The means of action hyper-fictions created for themselves on the basis of those much talked-about “technical innovations” of the digital world border fully on aesthetic matters (in much the same way as cinematic art is based on the cinematograph). Interactive, narrative spectacle, hyper-fictions and their means of action, are the culmination of a long semio-pragmatic history of fictional expression. That there should also be market, industry, “spectacularization”, etc., does not authorize us to neglect the totally new expressive agents developed in the digital world. Besides, one could point out that the invention of computation is a fully cultural process (tied to the history of the automaton) and that, from one invention to the other, interactive computation evolves in the context of a social redistribution, in a broad sense, of the relationships between producer and receiver.

E.A.A. : But for all that, doesn’t a new separation of the two terms forming the word spectactor turn out to be necessary so as to include the question of the interface ? This later joining together reception and intervention constitutes the environment and main means of a stage where the play of the spectactor opens up. Settling and adjusting the distance and implication phases, interface seems in that capacity comparable with a reactive musical score, which accentuates the relationship and orchestrates the condition permutations between those two opposite poles of action and observation.
Across it, the user finds him-self in an all the more fascinating place as it gets closer to a kind of paradoxical injunction. On the one hand, a promise addresses itself to him, that he will have the main role and a central

Page 241--------------------------------------------------------------------

responsibility, while on the other hand, he is left with the frustration of merely being able to watch the software directing her.
This ambivalent relationship tends to create a tension : to what extent should the user act and when should she put up with mere watching ? On the surface, the social gap distinctly separating the actors-on-stage and the public-in-the-theater seems to be closing up, but I believe it’s still there within the spectactor herself, interiorized by that “interactive subject” that has to make it on its own opposite to the computer.

J.L.W. : I agree with the ideas of “ambivalent relationship” and “frustration”. It’s the heart of the issue : the aesthetic and epistemological interest (at least in the field of Communication theories) of this between-the-two, where the subject is neither completely the “actor” of the spectacle (which designates either the interpreter’s or the director’s situation), nor the receiver in the usual sense of spectatorship (as in the spectator who contemplates, participates and recreates for herself the work played in front of her). For me, being spectactor is also being spectator. Nevertheless, to be a spectactor, you have to intervene in the formal matter of the propositions (stories, works, etc.). Formal matter, segments and links that is, without it being possible to isolate the segments from the sequences where interruption lies literally in the hands of the subject. An intervention to which the spectator did not use to be invited even in such cases - sculpture or architecture - where her own movements necessarily participated in the reception. In the interactive space, freedom is being tested as a limit. Interactivity is a play with limits ; it shifts, experiments on them, but does not do away with them ; hence the frustration. The promise (and the experience) of an augmented freedom whets the appetite and the satiety is always lacking. (That deception is sometimes the very source of artistic works, such as those by Jean - Louis Boissier, for example).

At the same time, the opposition between action and observation should itself be questioned. I remind you that that opposition is the very symbol of the interactivity question as set by the best representatives of the film critics (Daney, for instance : opposition between the image and the “visual”, which Debray takes up in “Life and Death of the Image” [3] as well as by a most important part of contemporary aesthetics (namely Lyotard : what is being aimed at today...is, on the contrary, that the one who receives does not receive, that one does not put up with being disconcerted, it is one’s auto-constitution as an active subject in regard to what is being addressed to it [4]). Let’s simplify their comments : action depends on technical operationality.
The stimulus-response model predominates. It concerns the accursed (animal ?) share of the human. Observation, contemplation mobilize the noble side, the one which introduces to the sublime (beyond the senses), to the intellect, to the sense of beauty (inseparable, if we follow Lyotard, from belonging to a community). Hence the emphasis on immobilization in the cinema theater as a spectatorial condition, contemplation in fine arts, the way of listening to music... I remember a discussion with Pierre Lévy, that took place a long time ago when he was still a technophobe, which concerned that opposition (see the book on “The Informatization of Society” we wrote back in 1985).

One of the merits of the interactive situation is namely the fact that it strongly questions, following Phenomenology (and Bergson, in particular), that partition of action - observation. In the heart of a particular type of action (a corporeal learning like a dance step), Bergson tells us, retention of memory (remember the engrammed figure) and anticipation (put oneself in a semi-distance of oneself in order to imagine-see-imitate the (learned) movement (to learn). Let’s establish a rapid link : the aspect of the interactive situation on which I concentrate my attention is precisely the one that unites action and contemplation, the one that prepares action through recollection of past actions and anticipation of possible effects (taking into consideration all the formal aspects such a successive formulation may include).

E.A.A. : Reminding in such a pertinent way the arguments put forward by the detractors of the state of existence and presence established by interactivity, makes me see, which is quite interesting and even amusing, that they are reversing the very connotations attributed to the actor and the spectator I was talking about in the beginning of our discussion. Nonetheless, beyond the debate on the relative values of action and observation, it is a kind of reaction stuck in the old dichotomy between mind and spirit that, in my opinion, arises once again. You have insisted on the fact that interactive systems spark off a growing implication of the body that in your latest work [5] you qualify as a growth trend of the body coefficient in tele-presence situations. I believe that that mutation continues the revolutions initiated by phenomenology and psychoanalysis, both of which radically contested that division of the mental and the corporeal. In this connection, the fact interactivity has become a privileged area for meeting and confrontation between a very wide range of disciplines and currents of thought, is quite revealing. Cognitive psychology, phenomenology, constructivist epistemology, artificial life, functionalism, and more recently, psychoanalysis, give indeed the impression of seeking an over-hanging legitimacy for their own foundations, as though a new definition and conception of the human subject were at stake.

J.L.W. : Interactivity’s interdisciplinary matters probably revive crucial debates, which, in fact, go far beyond the

Page 242--------------------------------------------------------------------

questioned subject. To return to the intervention - contemplation interweaving, it seems to me that the narrative sphere is one of those where it that interweaving proves to be the richest, for narrative tension offers its movement to sustain those action - observation exchanges. (That last remark can probably extend over the whole of the fictional sphere, but that remains to be demonstrated). Most of the times, interactive activities (navigating in a data site, discovering a virtual museum, etc.) also mobilize an action - observation compact but in a namely reflexive and / or cognitive dimension. I do not want to depreciate those operations on principle (spectacles also comprise a cognitive dimension but structured on the basis of perception and action). I am only trying to affirm the interactive spectacle specificity compared with the whole range of interactive modalities. This particular action is not pure perception (as is the case with spectacles) but also specifically, gesture.

E.A.A. : Let’s just remark that by physically concretizing a part of the mental work that every work imposes on its addressees, interactivity provides us at last with a system, which is capable of demonstrating the abstract processes inherent in every reception. Those processes used to be only theoretically accessible, like in the text-reader model, but are henceforth offering themselves, as the established relationship with the computer goes on, to experimental analysis.

J.L.W. : When you suggest, “a new separation of the two terms forming the word spectactor so as to include the issue of the interface ...”, I think you are reintroducing an opposition whereas the type of the relation is worth thinking through (it can as well be a percussion or an interweaving). You are probably right to mention cases of reflexive type (simulation piloting) where action and observation are practically indiscernible, but, in my opinion, it is on a different ground that one should be able to gather the nectar of interactivity. Proust wrote : “ In reality, every reader is, when she reads, her own reader. What needs to be discussed is not the supposed activity of the spectactor as opposed to the passivity, which is an absurd hypothesis, of the reader, the looker or the spectator, but the very nature of that activity. The question of the activity is the gesture here. The irruption of gesture in representations - and, through it, the irruption of a kind of form of corporeal presence - has almost gone unnoticed. And yet, it is the essential factor in the digital image world. Gesture is what makes the spectactor present in the spectacle. After the addition of sound to image, it is the second fundamental extension opening towards totally new stylistic forms and demanding a suitable vocabulary. One of the essential aspects of interface in general lies in the question : how to stage that call for gesture ? The underlying hypothesis being - we have been putting it to work in the seminar - that the gesture dimension mainly specifies interactive reception.

Hence the name “acted image” I have suggested - an image that causes and demands actions - synonym to interactive image but putting the emphasis precisely on the action in the sense of gesture.
This type of gesture is still a quite unexplored form of thought because of the great influence of a kind of non-interventional image culture (cinema, visual arts, etc.). It is this thinking of the gesture, or this thinking gesture, that interests us in our research, when displayed in fictional environments, in particular.

E.A.A. : Which reminds me that numerous superficial commentaries have insisted on a risk of “decorporealization”, pointing at an abnormal exit of our incarnated condition. When projected in a digital environment, which I prefer to call synthetic rather than “virtual”, the individual supposedly absents it-self so as to escape the world order. A mobilized, watchful body absorbed through a joystick or a keypad like an empty, automated envelope showing a disquieting transgression.

Hence the need for a seminar which reflects on gestures without judging them from an exterior point of view, as a sign of enslavement, but as revealing of an investment in a relationship where the body has an essential place. From cursors moving in an analogical way to avatars and simulacrums responding to a vast range of commands, we have examined the ways in which the interacting subject figures in the “spectact” [6] and how graphic forms symbolize her.

J.L.W. : That is where the question of the interface comes in and I would like to define it as a translating, transducing environment between the semiotic world of the interactive work and the necessary presence of the addressee (the spectactor) without which this world remains dumb, virtual, in the sense of “waiting to be activated”.

« Spectact » is the place to experiment on a kind of gesture aesthetics and gesture thinking that I call “interfaced gesture” because it is necessarily through an “interface” (mouse, button graphics, and beyond such objects, global semiotic organization of a world to perceive/act upon) that it occurs. There is no computer interface that does not presuppose that gesture dimension (most of the times a manual one), otherwise it is just a perceptive surface demanding no intervention at all. If the cursor is an extension of the hand, it is, in fact, an imaginary one. Its consequence is a transmutation of easily manipulated objects (the radio button that lights up when scrolled, for instance).

Gesture thus fabricates an imaginary relation, which is being concretized between body movement and the concerned object.
In that relation, the object (in fact, its image) signals its potential functionalities, or other facets, to the spectactor as if it was dashing towards her, whispering its

Page 243--------------------------------------------------------------------

favorite activation and animation modes. That is where the “interfaced gesture” concept comes from : it is the manual movement channeled by the mouse (or rather subjugated to the mouse constraints : analogical eye-controlled moving), which mutates in an visual exploration where the hand shows the way (like lighting the way with a torch). There is a short-circuit between what “interfaced gesture” (subjugated to the mouse constraints) designates and a way of considering a sole and the same new perception (and also a new thought in Deleuze’s sense when he talks about a cinema-specific kind of thought - and we can also invoke a thought of the text, of photography, etc.) peculiar to this bond gesture. “Bond“ (we could also call it extended, hybridized, channeled, organized...) expresses the constraints in their productive dimension.

Clicking would be the typical example of a totally new semiology, a mixture of natural and symbolic gestures immediately becoming meaningful in the stage space.
Unlike its value in the text space, clicking as far as the image is concerned owns a very vast repertory of significations since the later depends on the semiotic contexts in which it operates ; the pointer transforms into a viewfinder, then into a zoom, into various weapons in video games. And gestures on tactile interfaces adopt as many signifiers as action contexts (flying over, gripping, vibrations, movement, etc.).

Action on image through interposed interfaces includes indeed a considerable part of “unreal” since one does not act on the objects surrounding one in the same way one manipulates icon effigies. That gesture is a “symbolized” one (hand-conducted eye-mouse). Acted image remains, in that aspect, a visual representation (otherwise, one would be manipulating material objects, modeling clay or toy soldiers) ; it is then also “doped” with those “reality guaranties” as Metz calls them in his article on the sense of reality in the cinema theater, which are the elements of the “acted” signifier (designation, selection, controlled moving, sequence decision, all those codes that could be relegated as mastery but are counterbalanced by a number of disappointments and even of surprises. When addressed to image, gesture animates images of the world (“reality guaranties”) but according to often imaginary protocols : for example, wide range zooms, seeing through obstacles, instant jumps from one point of view to another.

Some command gesture codes have become “natural” at such an extent that, through those very artificial in reality devices, one sometimes feels like his body is actually moving inside that space (and we have gotten used to seeing with our hand) namely when one slides along a narrative flow.

E.A.A. : I believe the first consequence of the “interfaced gesture” to be a kinetic implication, that is a semiotic analysis of the corporeal implications engaged by an interactive system. A main preoccupation would be to question what a person mobilizing her body and senses in front of a computer or game platform does concretely. Consequently, it opens up to a gesture semantics the corollaries of which vary according to what software represents and/or simulates.

There would then be the observable, what is visible from outside, independently from what is on stage on the screen. And there would also be signification, symbolic and operational reach, and subjective interpretation of the interfaced gesture in the way it occurs and becomes meaningful for the interactor. These aspects depend on play narration, that is on the narrative world of reference, futuristic or even historic, and on the modes of participation permitting to access it and to take initiatives and achieve goals. Indeed, to the same interfaced gesture (moving the mouse, for instance) corresponds a multitude of interventions, which vary from guiding a digital character or navigating through a three-dimensional space to adjusting the cursor to the form of a target. In order to qualify in a specific way those operations, which develop in the interactive relationship and only make sense within it, I take the opportunity of our discussion to propose the term “interact”. It would qualify the significant actions as they occur in this very particular dimension of exchanges between a biologic intelligence and an algorithmic-logic machine. Those two confronting entities, as they are each endowed with reflexive abilities, support end carry every interact, which from each point of view acquires respectively distinct values without which an interact could not exist : cognitive, narrative, affective, act ional for humans or simply digital for the computer.

J.L.W. : Yes, it is an interesting notion. I would add that the work done those two last years in our seminar have often chosen video games as its target because we found it obvious that a gesture vocabulary was being elaborated there, through figure testing, and was piling up a whole arsenal infiltrating effectively the conceptual and practical set of tools of all the multimedia producers. It is in that domain (Tomb Raider, precisely) that I came across the idea of a possible relation between J.P.Eskenazy’s notion of the “imaginary body” in cinema [7]. Tomb Raider is not a work (like every video game with an end), but has fuelled a collective forge of expressive means available in different configurations (hyper-fictions like Sale Temps, Postales, Moments de Jean-Jacques Rousseau by Boissier...)

E.A.A. : Indeed, reusing the imaginary body concept, which came from the psychoanalytical field and then was adopted by film theory is very stimulating for someone like me who has studied film theory. I believe that, with this notion, surpassing the action - observation couple becomes possible.

Page 244--------------------------------------------------------------------

Unlike cinema’s imaginary body, which originates from the perspective point of the camera and is constructed, on the other side of the visible, through visual enunciation and monstration, interactive systems seem to me to be summoning two intimately related bodies that should be distinguished, which join together in a singular “interactive corporeality”.
On an operational level, a body we could call “projective acting body”, which means a body capable of acting according to the intervention modalities conferred on it by the interface : it moulds and invests itself, slides in the interactivity field. At first, it explores the potentials of influence offered and then it ends up fully employing them. For instance, in Tomb Raider, the whole question is exclusively about the simulated body of a woman, Lara Croft, and her properties, as it is the sole means and way possible to act within the play narration.

As a sine qua non condition of its efficiency, the projective acting imaginary body couples simultaneously with a sensitive and receptive dimension and a “perceptive body” which, for the spectactor, depends on :

what is given to her to feel by hearing, vision or even touch through joystick vibrations
the interpretation of felt percepts in connection with their instituting codes (motivated icons, significant graphics, realistic vision, etc.)
room for maneuver she knows or thinks having
lastly, the environment, diegetic or narrative, in which she is represented or implicated.
In the case of Tomb Raider, this perceptive body feeds on a complex association. On the one hand, the visual point of view of the pursuit, that camera automatically following the character all through the adventure, produces an imaginary body in Eskenazy’s cinematic sense, that is resulting from a primal identification to the device of “shooting”, here the simulated 3D perspective.
On theotherhand, the perception of Lara Croft’s body - in turn wounded, menaced, running, falling, fighting - comes closer to the cinematic secondary identification, the one felt towards the character according to the situations she goes through. In this case, secondary identification is based on situation and what’s more on constant evaluation of Lara’s action and reaction abilities.

Those two “corporeal” and mental entities stem from the meeting of a human subject with a computing object, with which she progressively undertakes to live a subjectively unstable and shifting relationship, which yet rests on constant principles, like visual pursuit, conducted body, as indicated in that brief analysis of Lara Croft. Dissymmetry between a sensitive dimension and an acting one, would contributeto the creation of a fascinating corporeal existence at the same time disabled and limited, but offering an illusion of power over everything. That last impression comes perhaps from the experiencing of a condition radically incommensurable to the trivial real world, and is ineffable as such. Different from one simulation game or digital world to another, this interactive corporeal existence would fan out from one software to the other detecting some major principles of the perceptive and projective body so as to shape itself through learning. From that angle, decomposition in phases of observation and action, even during a study determined to disentangle active passages from contemplative ones, seems no longer capable of reconstituting the wide-ranging nature this approach works out. By evacuating the essential interdependence of those two bodies, perceptive and projective, which are tangled, that study would lose any legitimacy, just like a hand, without the sense of touch, vision or proprioception, could not grab anything.
J.L.W. : This idea of a projective and perceptive body seems appealing ; perhaps, though, still caught in the action-perception dichotomy. In addition, I underline the fact that the suggested description of the perceptive body includes, quite rightly, an important cognitive aspect. We could or should perhaps, with extreme precaution, add another dimension to it, the aesthetic pleasure, which includes shock, destabilization and even deception. I notice our difficulty to exit carved perspectives. As soon as we invoke a divided body, we get suspicious against oversimplification and if we go no further than the ineffable singularity of the interactive experience, we have no set of expressive tools. The imaginary body notion offers indeed a productive perspective to get - partly - out of those difficulties. In cinema, says J. P. Eskenazy, we have to forget our own body so as to adopt an imaginary one, the one the film fabricates for us. If we cannot shape ourselves according to the corporeal proposition, we cannot understand the film.

Can the “imaginary body” notion function in a video game like Tomb Raider for example ? Yes, but only if some shifting is done. The classical puppets are, more or less, controlled by the puppeteer while here it is more about shared control. The imaginary body fabricated is not Lara Croft’s body. It is rather the body of someone who at the same time manipulates and guides her ; who observes the effect of her acts. It is a new imaginary body formula coming after the one cinema fabricated, within which one is at the same time assigned (projected in a posture ruled by the program and its constraints) and assigning (orienting that body according to the acts one accomplishes). If understanding a film is fabricating a memory of the film, here, with the “acted” imaginary body, it is more a question of intermediary fabrication between a registered will (the program) and a will governed by the player. Our imaginary body, thus set up, has to negotiate with that partly opaque character. The character is neither autonomous nor controlled. It is the configuration of the interface that gives form to this imaginary body of the spectact.

Page 245--------------------------------------------------------------------

In this case, the interface is at the same time the figurative element and the command instrument, a surface of vision and a kind of action that are absolutely interlocked.
Is it an inside or an outside point of view ? It is the very pet subject of cinematic analysis. At the cinema, one should at the same time see and be seen (thanks to that imaginary body). Here, it would no longer be a question of seeing, it would be more like acting and seeing oneself act. A very new situation for it is completely different from the way one sees oneself act in ordinary life. Merleau-ponty writes : “one is always on only one side of one’s body”. At the cinema, one starts to be a little on the other side but absorbed in an automatic will - in the series of the recorded images. With video games characters, we are, for the very first time, on both sides ; for manipulating a figurine is seeing oneself act in a way. And seeing oneself act is the condition for learning to play. Playing is fabricating an imaginary body by a sort of dilatation of our body which meets the figurine body : one sees oneself within, but not like in the movies, one sees oneself through a memory piling up of the effects of the attempted acts, successful or not.

[1] In French, spectateur and acteur. Beyond the apparent rhyme effect missing from the English version, it should also be noticed that the acteur is not equivalent of the actor, which mainly designates the one-who-acts-on-stage. Acteur includes actor but exceeds in that it describes the one who accomplishes an action in general. It is in that sense that the term is used in the text. (N.d.T.)

[2] This seminar takes place since November 1999 within the framework of the « Paragraphe » laboratory, Hypermedia Department, Paris 8 University. For the texts and acts of the seminar on line : http://hypermedia.univ-paris8.fr/seminaires/semaction/

[3] R.DEBRAY, Vie et Mort de l’ Image, Paris Seuil 1992

[4] J.F.LYOTARD, The Inhuman : Reflections on Time. Cambridge Polity ; Oxford Blackwell ; Stanford, Calif. Stanford University Press 1991, translation by G.Bennington and R.Bowlby of L’Inhumain : causeries sur le temps, Paris Galilée 1988.

[5] Présences à distance, L’Harmattan, 1999.

[6] In French, spectacte, from spectacle and acte. In English, the term becomes spectact, which designates the spectactor’s action. (N.d.T.)

[7] J.P.ESKENAZY, Film, perception, mémoire, Paris L’Harmattan 1994.