An Experimental Demonstration on How Sound and Music Can Change the Spirit of Film

THE ROLE OF SOUND AND MUSIC IN FILM AND COMPUTER GAMES: AN EXPERIMENTAL DEMONSTRATION OF HOW TWO DIFFERENT STUDIES OF SOUND AND MUSIC PRODUCTION CAN CHANGE THE SPIRIT OF A SHORT FILM. (Paper has been published at ATMM 2012 Conference on November 2012)

Tolga Tem*

Abstract:

“Films are fifty percent visual and fifty percent sound. Sometimes sound even overplays the visual” says the famous director David Lynch. Without sound, it is not easy to express the feelings in a film unless silence is purposely used for a certain need of expression. In real life, our visual perception works together with our auditory perception – this is how we experience the world. When recreating an environment in a film or computer game, we have a choice of using sound in a familiar way for more realism or in a delusive way to affect feelings.

Of course, designing a sound or composing music are individual arts, but when we talk about “Film or Game Audio” there is another artistic dimension and area of profession that we must be aware of: “Music Production and Sound Design for Visual Media”. Accompanying this paper is a film that has had its sound work reproduced in two different ways, which will demonstrate how  sound affects the visuals. More explanations about the related scenes of these movies can be found in this paper and the movies can be watched below.

The Role of Sound and Music in Film and Computer Games:

Generally, the visual arts need the support of sound to convey a moment correctly and in a more powerful or easier manner. Even without the use of sight or in the dark, the human brain can generate its own imagery by analyzing the sounds around it and by envisioning its own reality. This can be observed simply by allowing an audience to listen to a movie scene and then by asking them to describe how they visualize it in their imaginations. If the sound work is effective, it is possible to observe that the audience members will have similar visual descriptions of the scene that they listened to. Even the time or the era can be identified simply by listening to the sound effects or musical style. If we look at it from this standpoint, the effective usage of sound will perfectly match with the visuals to help convey the overall feeling. This successful association of sound and visual elements will immerse the observer in a realistic environment while watching a movie or playing a computer game.

We should approach the subject with a question such as: what is the role of sound in film or computer games? Before we answer this question, it’s better  to review the basic information about the usage and specifications of sound in cinema.

• The types of sound in cinema: e.g. natural or unnatural sounds.
• The usage of sound: e.g. in the background or foreground, as a sound bridge, as a realistic or unrealistic source, or  for dramatic impact.
• The usage of music: music can be used as a natural (featured) sound source (like coming from a piano in the scene) or as a non diegetic sound (like the music only the audience hears but not the characters in the scene).
• Sound creates a sense of reality: without sound, feelings are negatively affected.
• Sound adds another dimension to vision: sound can even create a virtual environment itself. (An experiment can be done with the audience by asking them to visualize the sounds they hear.)
• Sound supports the concept: sometimes even silence can be a perfect tool to support the concept or can be used to create a contrast.
• Additional support of sound for visual media: delivering or adding the information easier, delivering the feeling of the moment, creating the rhythmic structure.

(Sözen, 2003:184)

To demonstrate the role of sound and music over visual media, a special audio/video production, a short film called “TAXI”, has been produced. The two versions of the film (each one is 6 minutes long) use different sound production to emphasize “how sound or music may change the spirit of the visual expressions”. This film was created using the video editor in the computer game  “Grand Theft Auto IV” (GTA IV). Later, the sound channels were reproduced using additional sound effects and different styles of music as well as narrations. As a result,  audiences watch both a happy and sad version of the same film and become more aware of “how the sound affects the visual”. Additionally, this movie includes most of the roles and cinematic usage of sound as explained in detail below.

Analysis of the Usage of Sound in the short film “TAXI”:

Music as non-diegetic sound:[1] The film begins with an aerial view of New York, with different music in each film triggering separate emotions. While the song “New York, New York” by Frank Sinatra introduces us to the joyful side of the city in the happy version, another song (which was specially composed for this production [2]) prepares us for a sad story from the very first moment. A good composer should know how to play with the notes and the chords well enough to be able to speak to the human heart. This is very important in film and is also the crucial point in film scoring.

Narration: The story teller (who is actually the protagonist) begins telling us his story, which conveys his own emotions to the audience. The theatrical usage of voice is very important, as its effect gets us into the mood of the story. While he speaks with an energetic and lively tone in the happy version, his sad and hopeless tone can easily be identified in the sad. We begin to discover what he thinks about living in New York as he slowly starts putting us in his shoes. In the happy version he describes other citizens in an optimistic manner, and even while talking about the rough parts of the city, he believes that they have to be accepted as a part of the city’s soul. Of course, the differences in script also affect the flow but this is still in the domain of sound as it is being carried by voice over.

Off-Voice: There are times when we hear voices apart from the main character’s, such as radio announcements or other people interacting with each other. These kinds of sounds let us communicate unconsciously with the emotions needed. For example, in the happy version, while the camera is moving inside the house (01:45), we hear the radio announce a listener’s request for the song “Singin’ in the Rain” and we hear our character also humming the song to himself,  allowing us to share in his joy. In the second video, we hear the news reporter telling us about how many people are suffering from hunger or poverty, which our main character is also dealing with. The sound from the radio, the character’s voice, and the music climb in a crescendo[3].  Other examples include the woman on the street  speaking on the phone (03:30) or the speech of a passenger in the taxi(04:02), which when delivered with different voice overs convey the different emotions in the different scripts.

Music as the natural sound (featured music): There are times we hear the music as a natural sound source (the music is heard not only by the audience, but by characters in the film), for example, the music playing in the taxi at (03:06). This music is different in the two versions. In the happy version, we hear rock music which is well-suited to the character’s mood. In the sad version, there is a folk-song which is well known for its agitating and complaining lyrics. The choice of music here conforms to the mood very well.

Other Examples of the Cinematic Usage of Sound and Music in “TAXI”:

• Usage of music as background:  In the introduction, music is intentionally used without any other sound effects (such as wind or city sounds) for more dramatic impact.

(00:01) intro music.

• Sounds as a Sound Bridge: “Sound bridges can lead in or out of a scene. They can occur at the beginning of one scene when the sound from the previous scene carries over briefly before the sound from the new scene begins. Alternatively, they can occur at the end of a scene, when the sound from the next scene is heard before the image appears on the screen.” <www.classes.yale.edu/film-analysis/>

(00:58) Street sounds are heard before the camera moves to the related scene.

(00:43/happy version) Whistling starts earlier than the next scene.

(04:28) Music of the next scene is heard, before the transition starts.

• Changing the state of music (from background to featured music)

(01:43/happy version): The song “New York, New York” which plays as background music that the characters in the film don’t hear, gradually emerges as music coming from the radio. In this way, its state changes from being outside the scene to being inside of it. This effect is accomplished using frequency filtering that mimics the tone of a radio.

• Sound Perspective: “The sense of a sound’s position in space, yielded by volume, timbre, pitch, and, in stereophonic reproduction systems, binaural information”. <www.classes.yale.edu/film-analysis/>

(02:20/happy version) The music from the radio is heard clearly while the camera is close to it in the room, but when the camera jumps out of the room, we hear the music coming through the walls. This effect was accomplished by changing the volume and the related frequencies.

(02:32/happy version) The character’s speech and the call to the hot dog vendor were changed relative to the distance from the camera. If live sound recording is not preferred after shooting (or if it’s an animation or game as in this example), this effect is processed in the recorded sounds in a studio (03:05-03:38). The perspective of the radio music sound changes according to the camera’s position several times. (inside/outside the car, passing by etc.)

• Implying events using sound: Using the sound primarily according to the visuals. (Sözen, 2003:204)

(05:25-05:34/happy version) Related sound effects give information about what the character is doing while he is not in the frame.

(05:48-06:22/sad version) During the closing credits, we hear some sound effects that allow us to imagine what’s going on although we can’t see it. It is possible to understand that our character has leapt from the bridge by the screaming of bystanders. He was on the verge of drowning in the sea, but after being under water briefly, we can hear  his breathing and therefore his survival. The sound of heartbeats supports his being alive. There is also sound perspective used here when he goes under water and emerges a couple of times. The entire scene is conveyed using only sound. This is a perfect example of the visualization of sound.

CONCLUSION: 

This short film project demonstrates some of the roles of sound and music and how they affect the visuals. It helps to illustrate the importance of sound over visuals, and how sound can create its own imagery even without visuals. Therefore, as a sound designer or composer, it is very important to know how to produce the appropriate sounds for the visuals, and how to compose for a scene when producing films or game projects.

REFERENCES:

Sözen,Mustafa: Sinemada Ses Kullanımı, Ankara,Detay Yayıncılık,2003. Print

Mutlu, Erol: İletişim Sözlüğü, Ankara, Bilimsel ve Sanat Yayınları,1998. Print

Yale University,Film Studies Program,New Haven, CT, 2002. Archieve.Web.

Film Sound, Leaning Space dedicated to the Art and Analyses of Film Sound Design, Archieve. Web. 2012. <www.filmsound.org>