By: Venzha/Hasan Murod

Abstract:

Sonification is a new technique to represent data by means of sound. There’s many success stories of sonification  in many fields of application to improve human’s understanding of the environment. In the art field, sonification need to be explored  as an artwork creation tool. Movement sonification is the main exploration in  my work titled Artificial Intelligence Part I, presented in Jogja Biennale 2005, Yogyakarta, Indonesia. In my exploration, sonification can be used as a creation tool to produce an artwork that has an interactivity with the audiens and the environment, thus help us to get better understanding of our environment.

Introduction

Sonification is a new technique to represent data by means of sound, forms an interesting alternative to visualization techniques. Sonification is defined as the use of nonspeech audio to convey information. More specifically, sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation . By its very nature, sonification is interdisciplinary, integrating concepts from human perception, acoustics, design, the arts, and engineering.

Naturally sonic environment gives us better understanding of our whole environment. Our sonic perception of a bumping car give us an understanding about how hard the bumping happens, and it’s more sensible than our visual perception. With sonification,  we can hear anything that usually doesn’t generate sound, thus give us better understanding of our environment.

There’s many success stories in many fields of application in improving human’s understanding of the environment. Geiger counter invented by Hans Geiger in early 1900 use sonification technique to give auditory display informing radiation level to the user. The Pulse-oximeter became standard equipment in surgical operation in the mid 1980s in the United States, it generates a tone that varies in pitch according to the level of oxygen in a patient’s blood. Lunney and Morrison (1990) reported a success in presenting infrared spectrometry data using sonification for blind chemists and chemistry students.

Movement Sonification, An Exploration of Sonification as An Artwork Creation Tool

In the technical field, sonification is still considered to be in it’s infancy. In the art field, sonification need to be explored  as an artwork creation tool. Movement sonification is the main exploration in  my work titled Artificial Intelligence Part I, presented in Jogja Biennale 2005, Yogyakarta, Indonesia. The goal of the work was to build a preliminary artwork that can be advanced to produce a more intelligent artwork in the future. This preliminary artwork consist of hardware and software for capturing and processing a real-time image and generating a visual and sonic artwork based on the  image.

The Mechanism

The implementation of the work is done using a web camera, a laptop, a multimedia projector, and a sound system as the hardware.

The camera is used to capture the image of the workspace in real-time. By analyzing  frame-by-frame of the images, the movement’s features of the captured object is then extracted. The features consist of the  position, direction, and the velocity of the moving object. Other image’s fatures  that are not movement-related such as maximum intensity position  and “black area”-to-“white area” ratio also extracted. The datas of extracted features is then fed into sound generator to produce the sonic artwork. The sound generator consist of twelve stereo oscillators. The frequency, volume, phase, and the  balance of each  oscillator is controlled by the data from extracted features. The volume of the whole oscillator is controlled by movement detection mechanism. If there is no movement of the object in the image,  the volume of the whole oscillator is then faded out in a few seconds.

The Aesthetic Aspect

The pitch generated by each of twelve oscillator directly and independently follow the audiens movement’s feature, thus the melody flow continuously, not matched to a predetermined scale or pattern. Their combination form either consonant or dissonant harmonics, giving a rich experiences to the audiens. A various timbre is created by complex interaction in the mixing of the oscillators’s output, including superposition of two adjacent frequency that generate beat modulation. Some oscillators are programmed to generate a tone with a random frequency to improve the dynamic behavior. Because the randomness is controlled by the speed of movement, the audiens can feel a more dynamic “composition” as they move faster. The control of the balance and phase of each oscillator improves audiens’s experience of moving in a space dimension.

The Benefit to The Human Understanding of The Environment

In the technical field, sonification gain a great  success in improving understanding of our environment. In that field, sonification is very helpful in understanding many real entity existing in our environment. My exploration in the art field show that movement sonification can also improve audiens’s understanding of some abstract entity existing in the environment of the workspace. The interactivity between the work, audiens, and the environment gives a chance to the audiens to feel their movement, their acceleration, their position on the workspace,  and their interaction with other audiens by means of sound. In physics, the audiens may have learnt the abstract concepts of movement, acceleration,  and position, but here, with movement sonification, the audiens are invited to feel them like  percepting  real entities.

Conclusion and The Future Work

In the technical field, sonification has been success to improve our understanding of our environment, and in the art field, especially in my exploration, sonification can be used as a creation tool to produce an artwork that has an interactivity with the audiens and the environment, thus help us to get better understanding of our environment.

Our future work will focus on advancing the work to produce more intelligent artwork by implementing artificial intelligences  in the image processing, sound generation, and the visual generation. By means of artificial intelligence, we hope that the artwork will have more environment-aware and audiens-aware, giving more interactivity between the artwork, audiens, and the environment.

References

• Kramer, Gegory, et al, 1997, Sonification Report: Status of the Field and Research Agenda.  (http://www.icad.org/websiteV2.0/References/nsf.html).
• Lunney, D., & Morrison, R. (1990). High technology laboratory aids for visually handicapped chemistry students. Journal of Chemistry Education 58, 228.