SOUND DESIGN IN MEDIA

The art of desynchronization by Jean-Jacques Birgé
© A.P.R.E. 2004

Talking of media, we are of course evoking audiovisual media.

Audio, the first part of this two-components adjective, has too often being ignored or forgotten.

Most of the time, sound is mistakenly considered as post-production, rather than being thought of, as soon as the writing process begins, and its budget remains too small a part of the total budget.

What is true for cinema is alas also true for multimedia, as they both belong to the same history of audiovisual, which began before 1895, with Émile Reynaud, Thomas Edison and the Lumière Brothers. So, what has been discovered for movies during the last century should also be used to write and produce new media projects.

At the beginning of the history of cinema, films were silent. In fact, they were never shown without sound. Even in small theaters, there was always an orchestra, a pianist or another soloist, sometimes a narrator or sound effects, at least a Gramophone (in 1976, I had the idea to play live accompanying  silent movies with my own orchestra, Un Drame Musical Instantané). Intertitles could eventually help the understanding, but during that period cinema had to be very inventive to tell stories without words. At the beginning of the 30s, movies became talkies. Quite a catastrophe! For the next decades, only a few directors have understood the fantastic power of sound.

From Fritz Lang to Jean-Luc Godard, from Jacques Tourneur to Luis Bunuel, from Jacques Tati to David Lynch, some of them have sought ways to use sound not as an illustrative effect but as a complementary element to pictures. At the beginning of Das Testament des Dr Mabuse by Fritz Lang, the music of the credits becomes a loud repetitive machine that smothers all other sounds and dialogue, so you have to guess what is said, just watching the action. Suspense and fear! In M, the Grieg’s theme from Peer Gynt, whistled by the murderer, is the motor of the screenplay. In La femme mariée, Jean-Luc Godard, shows Macha Méril reading a magazine in a cafe, while two other girls discuss sex it at another table. Godard questions the normal mixing of sounds that normally privilege dialogue. In Lola Montes, Max Ophüls uses echoed dialogue to go back into the past. In Sur mes lèvres, Jacques Audiard lets us hear another world, the one heard by the deaf girl, and uses this handicap to run his story…

Sound can widen the space of the screen (off-stage), making us guess what is not shown. Listen to the landscape or the ambience, during a close-up: sound might suggest a different time or space from what the picture shows. The edges of the screen become the border between sounds and pictures. The man on the close-up might not only be in a place you don’t see but hear, he might imagine another place, another time, another situation. At the beginning of Psycho, Alfred Hitchcock shows Janet Leigh imagining what is supposed to happen in the place she has left after her robbery. And specially on TV, why should we have to show the one who speaks rather than the one who listens? It becomes a question of screenplay, a question of language.

Sound suggests more than it shows.

Robert Bresson said that a sound always calls a picture up, a picture never calls a sound up!

For a sound designer, it is not what you show which is important, but what you suggest. I always remember Godard explaining, about the editing, that « it is what you take off which is important, not what you keep. » As the example of the edges of the screen, he points to the cut, the ellipse, a no man’s land, which neither belongs to one shot nor to the other.

A sound illustrator makes the sound of the action we see on the screen. A sound designer works on its color, so that this sound becomes funny, sad, scary, interrogative, it plays with consonances and dissonances to create dramatic effects, producing any other emotion from desire to anger, from lightness to darkness, or giving clues to what has been happening or to what is going to come… In Raging Bull, Martin Scorcese uses animal sounds instead of the real voices of the boxers. In his sound films, Jean Epstein invents the sound close-up, just slowing up sound files. In Lancelot, Bresson seems to use only one track at once, mixing all sounds at the same level, metal clothes and footsteps acting as the rhymes of a poem, and blood flowing out a decapitated body as if it was a river. In all his films, Mizoguchi Kenji mixes sound effects and music as if they were all belonging to the same score. For Alain Robbe-Grillet’s films, Michel Fano conceived all the sounds of a soundtrack as all belonging to only one score, he called it « sound score ». Listen to David Lynch’s movies, or to the soundtrack of Amélie Poulain!

A sound score and a sound chart.

There should be a sound chart as there is a color chart. All the sounds of an audiovisual project might obey to rules the director could chose with the help of a sound designer, just as he works on lights and colors with a director of photography. It would give homogeneity, shape and style to the film. If there is often a sound designer on an American film, it still remains very rare in France.

So, voices, noises, ambiences, music all belong to the sound score, and the choice of them, the way to record and treat the sounds, to edit and mix them with the pictures, is part of the art.

Even if the color of a voice might be very important, the sound designer merely never interferes on the casting!

Working on sounds, we usually class them in two categories, short ones (effects) and long ones (ambiences). It’s more convenient, but it also deals with the difference between action and situation.

About music, I discovered very early that any piece might fit any picture. But the meaning of the result might be very different! Then the role of sound design is to control that meaning, following the needs of the screenplay.

There is also an eternal question about using preexistent or original music.

The interest and drawback of preexistent music is that it brings cultural references: the first movement of the fifth symphony by Beethoven in Verboten by Samuel Fuller, Strauss’ Waltz in 2001, a space odyssey by Stanley Kubrick, or anywhere the Wedding March by Felix Mendelssohn! But be careful with individual references: a piece, which suggests a beautiful story to someone, might have an opposite effect to someone else (for example, if you fell in love or broke up while listening to a certain piece). So it depends if the reference is general or personal. And be very careful of the cost of the rights, which can ruin a production (it always belongs to the producer who paid for the recording!).

Original music is not more expensive, and it often fits better the purpose, the timing, and the color of the whole score.

Under certain circumstances, music might even help the shooting, if it has been composed before it, following the practice of D.W.Griffith, Michael Powell, Jacques Rivette and many others.

There is nothing worse than video-clips where pictures and music are totally synchronous. What’s the use of this military hypnosis? In most long feature movies, it is also very common to have mellow strings on love scenes and speedy punchy rhythms on action! What a bore!

Very often, I would prefer to choose to follow the concept of accidental synchronism invented by Jean Cocteau in Beauty and the Beast. Cocteau had asked Georges Auric to compose music for different parts of the film, and then he exchanged the parts and put music of one scene on another one, etc. Magic then appears, because, if it fits the purpose, it plays with anticipation, delay or echo effects. There is no need for synchronism unless you need a special effect such as suspense, coup de theatre, etc.

In the same way, when I compose music for audiovisual medias, I first think of the general ideas, the meanings, and then later, having memorized the pictures, I compose following the timing, and almost never watching the pictures while I’m recording. Music obeys time rules that could be destroyed by any tight screen timing. I noticed it fits better than if I had decided to follow each breath, each movement, because if the purpose is right, new synchronism appears, with magic effects no one could have guessed before. Afterwards, you can always try to move synchronization, and be amazed how the result might change if you even try a very short delay such as one or two frames. Synchronism is a decoy. Sound design has nothing to do with it. If I dare, I might sum up saying that on one hand it’s a question of color and esthetics, but on the other hand it’s the art of desynchronization!

Sound in multimedia follows the same rules as in linear medias such as TV or cinema, but a few parameters are specific, due to technical needs, and more important and interesting, to the possibilities offered by interactivity.

Off-line products such as CD-Roms or on-line projects on Internet are submitted to questions of size of the files, number of tracks, speed of the processors or loading time. This leads us to deliver each sound separately, to build it as short as possible, to make loops rather than long evolving ambiences, and so, to compose on that special purpose. But such constraints might also drive us to new ways of thinking and composing.

Technical problems are not the most interesting thing to discuss, Director Jean Renoir used to say that we need to know technique in order to forget it. Depending on the projects, I use acoustic and electronic instruments, natural sounds and some I do with my mouth, computer treatments and real time recordings, sequencers and sound programs, reporting and studio, etc. Once again, it all depends on what is the purpose of the project. Each time I start to work on a new one, I begin by finding the right orchestration (sound chart) and this is not arbitrary, it is always imposed by the nature of the project.

The main question in interactive medias is to give life to machines. Machines are absolutely slavish; they never revolt themselves (bugs are not their choice!). Only humanity can incorporate mistakes to make art. Errare humanum est! If an artist is perfectly following all the rules that are being taught at schools, then his art may only be academic. His errors make his style. Berlioz is a self-taught guitarist who does not know much about orchestration and so he invents new ways of orchestrating, Mahler’s symphonies are too long and that’s fine, Apollinaire imitates badly Anatole France and creates Alcools, etc. Just compare a piece played by live musicians or programmed on a sequencer!

This is how I have been convinced to integrate errors in the system, in other words to get humanity into it. I decided to put three sounds better than one for any action and play these three randomly. I chose to make slight variations, or sometimes radical ones, when we come back to a scene we’ve already been into before, as time has been passing by. I found ways to build loops you could stand a long time, making them very common but adding random sounds upon them. Any sound might be considered as a beginning and an end, as well as belonging to the whole corpus. Etc. I have built so many new rules that I would now need to transgress them…

Considering all sounds (for example, there are 1500 sound files in CD-Rom Alphabet) as a whole score, working on interactivity in such a way that there is no discontinuity all along the discovery of that kind of object (see my first work on a CD-Rom with At the circus with Seurat), being fond of all new technologies, I was led to conceive, compose and record interactive music. Sound design and music composition are very alike to me, as I have very early adopted the Varesian way of conceiving music composition as the organization of sounds.

So, on one hand I only deliver sound files to the programmer I am working with, and on the other hand I fix composition principles. This collaboration is a necessity. All great programmers I’ve been working with are engineers, mathematics geniuses and great creators: without Antoine Schmitt, Frédéric Durieu, Xavier Boissarie, I could not have done what I did. Every time, I have to translate my feelings, ideas, concepts, tunes, harmony, into words, so they can translate them into algorithms. It’s a back and forth movement between us. Many mails, many phone calls, many days and nights of fun. At the end it is supposed to sound as I dreamed of it!

It really began with letter L in Alphabet, a string trio anyone could play his/her own way. It went on with Frédéric on the Internet site LeCielEstBleu.com, with the animals of our Zoo, the three musical modules of Time, and our last infernal machine, a strange music box called La Pâte à Son. I also would like to quote the work with painter Nicolas Clauss (he is also programming on Director) on Internet site flyingpuppet.com and somnambules.net, with whom I prepare art installations and live shows.

On the DVD, interactive modules: All Somnambules.net, and a few works with LeCielEstBleu.com
Actually at Centre Pompidou (Atelier des enfants): La Pâte à Son
Internet sites: www.drame.org, www.lecielestbleu.com, www.flyingpuppet.com, www.somnambules.net