Sound and music computing

The term Sound and Music Computing (SMC) refers to an emerging multidisciplinary research field. By definition, SMC research approaches the whole sound and music communication chain from a multidisciplinary point of view. By combining scientiﬁc, technological and artistic methodologies it aims at understanding, modeling and generating sound and music through computational approaches.

Detailed Exposition of the Field

SMC research can be traced back to the 1950's, when a handful of composers, together with engineers and scientists, began exploring the use of the new digital technologies for the creation of new music and multimedia content. This new field of research had a profound impact on the development of culture and technology in our postindustrial society. Since then SMC has not only made great advances in a variety of areas that range from digital signal processing to cognitive musicology, but has also contributed to many successful technological applications, ranging from the synthesis engines of digital musical instruments to the audio CD, MP3 and polyphonic ring tones.

The central focus of the research ﬁeld is sound and music. Sound is the resonance of objects and materials that we can listen to. Music is the intended organization of sounds for particular uses in social and cultural contexts. The sound and music communication chain covers all aspects of the relationship between sonic energy and meaningful information, both from sound to sense (as in musical content extraction or perception), and from sense to sound (as in music composition or sound synthesis). This definition is generally considered to include all types of sounds and human communication processes except speech. Speech research has its own aims and methodologies and is outside the SMC field.

The other elements contained in the definition statement above can be briefly explained as follows:

The multidisciplinary point of view relates to the use of various research methodologies and disciplines from the natural and human sciences. SMC also includes various research goals and approaches that deal with cross-modality, such as the relationship between perception and action and the integration of different senses involved in human-machine interaction (hearing, vision, movement, haptics, etc.), both in individual and social contexts.

Scientific and technological methodologies refer to empirically-based and modelling-based approaches that draw upon advanced tools for measuring and processing information. Artistic methodologies refer to approaches that explore human experience and expression.

Understanding refers to our knowledge of the mechanisms that underlie how people deal with sound and music in terms of its content and experience.

Modeling refers to the representation of knowledge through algorithms and tools.The resulting models are used both in applications that aim at scientific understanding (e.g. simulation of perceptual processes) and also in applications that aim at practical understanding (e.g. in sound-aware objects, music information retrieval systems, music production companions).

Production refers to the creative use of algorithms and tools to develop new content in which sound and music are communicated, as in sound environments, interactive artistic works and sonic design.

Computational approaches refer to the core processing which allows the development of tools linking sonic energy with subjective experience. Computing is the shared practice that connects scientific understanding, the development of technological equipment and content-based creation.

Disciplines involved

The disciplines involved in SMC cover both human and natural sciences. Its core academic subjects relate to music (composition, performance, musicology), science and technology (physics, mathematics, engineering) and psychology (including psychoacoustics, experimental psychology and neurosciences).

Music

As the name itself indicates, Music provides the core investigation area for SMC. It supplies both an endless material (both in scope and depth) for analytical investigation (Musicology) and the requirements to extend expressive means of creation (Music Composition), while Music Performance yields interest on both sides (analytical and expressive).

Music Composition: This concerns all research that has a focus on musical content creation and sound design. It includes creating music as a score, as an interactive artistic event, as a sound installation, as a soundtrack, and as any form of organised sound event which communicates information.

Music Performance: Music performance is a complex activity involving physical, acoustic, physiological, psychological, social and artistic issues. At the same time, it is also a deeply human activity, relating to emotional as well as cognitive and artistic categories. The research in this ﬁeld can be seen as ranging from studies aimed at understanding expressive performance to attempts at modelling aspects of performance in a formal, quantitative and predictive way.

Musicology: All research that deals with musical meaning formation, musical content description, and associated mediation technologies in particular sociocultural contexts. It comprises the study of how musical content can be described and how the subjective and sociocultural background of users plays a role in the production, distribution and consumption of music.

Science and Technology

Due to the interdisciplinary and problem oriented character of SMC, many disciplines in the scientiﬁc and technological ﬁeld are needed. Physics and mathematics deal with the physical vs. abstract side of SMC, while engineering disciplines face the technological needs to build systems and develop applications.

Physics: Acoustics is the science concerned with the production, control, transmission, reception and effects of sound as a physical phenomenon. The branch of acoustics of special interest to the SMC community is music acoustics. It includes the acoustics of musical signals (such as in expressive music performance), musical instruments and singing voices.

Mathematics: This area is concerned with the mathematical approaches to musical structures and processes, including mathematical investigations into music–theoretic or compositional issues as well as mathematically motivated analyses of musical works or performances.

Engineering: This includes all the research in computer science and engineering, signal processing and electronics that deals with sound and music information representation, processing and communication. It comprises multimedia information systems, artificial intelligence, audio signal processing, robotics, sensors and interface technology.

Psychology: This includes all research into music–related behaviour and brain processes, including the roles of perception, cognition, emotion and motor activities. Psychology is here understood to cover the whole domain from psychoacoustics to experimental psychology to neurosciences.

Areas of Application

SMC research can also be ﬁnalised towards applications that can acquire an important role in the definition of the ﬁeld. Current areas of application include digital music instruments, music production, music information retrieval, digital music libraries, interactive multimedia systems, auditory interfaces and augmented action and perception (e.g. games, interactive artifacts, and appliances).

Digital music instruments: This application focuses on musical sound generation and processing devices. It encompasses simulation of traditional instruments, transformation of sound in recording studios or at live performances and musical interfaces for augmented or collaborative instruments.

Music production: This application domain focuses on technologies and tools for music composition. Applications range from music modelling and generation to tools for music post–production and audio editing.

Music information retrieval: This application domain focuses on retrieval technologies for music (both audio and symbolic data). Applications range from music audio–identiﬁcation and broadcast monitoring to higher–level semantic descriptions and all associated tools for search and retrieval.

Digital music libraries: This application places particular emphasis on preservation, conservation and archiving and the integration of musical audio content and meta–data descriptions, with a focus on ﬂexible access. Applications range from large distributed libraries to mobile access platforms.

Interactive multimedia systems: These are for use in everyday appliances and in artistic and entertainment applications. They aim to facilitate music–related human–machine interaction involving various modalities of action and perception (e.g. auditory, visual, olfactory, tactile, haptic, and all kinds of body movements) which can be captured through the use of audio/visual, kinematic and bioparametric (skin conduction, temperature) devices.

Auditory interfaces: These include all applications where non–verbal sound is employed in the communication channel between the user and the computing device. Auditory displays are used in applications and objects that require monitoring of some type of information. Soniﬁcation is used as a method for data display in a wide range of application domains where auditory inspection, analysis and summarisation can be more efficient than traditional visual display.

Augmented action and perception: This refers to tools that increase the normal action and perception capabilities of humans. The system adds virtual information to a user’s sensory perception by merging real images, sounds, and haptic sensation with virtual ones. This has the effect of augmenting the user’s sense of presence, and of making possible a symbiosis between her view of the world and the computer interface. Possible applications are in the medical domain, manufacturing and repair, entertainment, annotation and visualization, and robot tele-operation.

Context

This part addresses the various contexts that determine how the research field of sound and music computing is embedded in a societal framework. Four of these may be identified, namely, the research context, the educational context, the industrial context and the socio-cultural context. The research context is about the state and trends of related scientific and technological developments and their influence on sound and music computing. The higher education context is about the education of future researchers in the field. The industrial context is about the impact on the industries and about the relevant trends in the information and communication technology (ICT) sector. Finally, the socio-cultural context is about the link to culture and the relevant social implications. These four contexts thus provide the background in which the state-of-the-art and challenges of sound and music research are situated.

Challenges and Strategies

Five broad challenges are identified which are of relevance to SMC research. The first two are centred on the actual research issues, the third one addresses educational aspects, the fourth one focuses on knowledge transfer and the last one is centred on social concerns.

Design better sound objects and environments: The growing abundance of electronically generated sounds in our environment, coupled with the rapid advances in information and sensor technology, present SMC with unprecedented research challenges, but also opportunities to contribute to improving our audible world.
Understand, model, and improve human interaction with sound and music: The human relation with sound and music is not just a perceptual and cognitive phenomenon: it is also a personal, bodily, emotional, and social experience. The better understanding of this relation from all these perspectives will bring truly useful and rewarding machine-mediated sonic environments and services.
Train multidisciplinary researchers in a multicultural society: SMC is a highly multidisciplinary domain that requires special expertise. But the way the established academic disciplines are being taught makes it difficult to acquire the proper knowledge. Thus there is a need for the establishment of appropriate educational programmes for training these specialists.
Improve knowledge transfer: A large part of SMC research is devoted to applications that can be directly exploited in the arts, in industry and in society at large. Proper knowledge transfer should result into an impact much larger than the current one.
Address social concerns: The role of the SMC field goes beyond that of a mere provider of technological or commercial solutions. SMC has the potential to contribute to maintaining and furthering the richness of human culture and preventing the global technological trends that make the world uniform. Also, SMC should empower users, putting the relevant choices and decisions back into the hands of the individual.

SMC Network

The SMCNetwork.org is a web portal hosted by the Music Technology Group of the Universitat Pompeu Fabra of Barcelona as a service to the whole SMC community. The portal is kept up-to-date with the SMC research activity. It also includes an annual SMC conference and a summer school.

References

Camurri, A., De Poli, G., and Rocchesso, D. (1995). A taxonomy for Sound and Music Computing. Computer Music Journal, 19(2):4–5.
Coulter (ed.), N. (1998). ACM computing classification system (1998). http://www.acm.org/class/1998.
Version 1.0 of the Roadmap: The S2S2 Consortium (2007). A Roadmap for Sound and Music Computing. Version 1.0. ISBN: 978-9-08-118961-3