☰

Musical Mechatronics (Writing from my PhD Thesis)

It is worth briefly mentioning the field of themed entertainment which in the 20th and 21st centuries have used musical mechatronics to create immersive entertainment experiences. A pioneer in this field is Disney with their The Enchanted Tiki Room in 1963 and a life-sized audio-animatronic Abraham Lincoln exhibited at the 1964-65 World Fair serving as two historically significant examples. While primarily concerned with attracting visitors to theme parks and other attractions for monetary gain, in the 21st century mechatronics have been used with themed entertainment with a greater emphasis on the artistic-implications of this methodology. Among the companies working within this paradigm is MEOW WOLF, whose interactive exhibits, including House of Eternal Return (2016) and Convergence Station (2021) present visually pleasing, sonically engaging, and artistically intriguing works that, according to MEOW WOLF, “inspire creativity through art, exploration, and play”. While the remainder of this genus-specific literature review focuses on the use of musical mechatronics within outdoor sonic installation artworks, the immersive experiences created by organisations such as Disney and MEOW WOLF have exposed massive numbers of visitors to mechatronic-facilitated works and therefore provide an important common cultural reference and possible conduit for exposing people to the field of artistically-inspired and -motivated music mechatronic projects.

Since the pioneering work of Trimpin and Godfried-Willem Raes in the 1970s, musical mechatronics has served as a niche medium for a growing number of installation artists, academic researchers, and performers. In recent years, academic discourse concerning musical mechatronics has investigated a wide range of topics such as incorporating extended performance techniques into mechanical instrument design, increasing the parametric complexity of instruments, improving mechatronic musicianship, and re-purposing ‘found’ objects into mechatronic sound objects. To maintain focus within this broad and varied discipline, this review focuses on two sub-interests within mechatronic installation art and performance that align with the Acropolis family design considerations: modular design methodologies and outdoor installation. To contextualise the Explorator project’s genus-specific research objectives within a lineage of historical and contemporary works, our overview begins with a focus on modular design principles within music mechatronics.

Modular Musical Mechatronics

Due to the inherent characteristics of these types of systems, most newly created mechatronic instruments, including those built for the themed entertainment industry and those exhibited for artistic purposes, are not designed with cross-project modularity as a priority. Instead, newly designed musical mechatronics are usually purpose-built to serve the specific needs of a particular installation scenario. Achim Wollscheid’s clapper system (1993-1998) provides an early exception as it is a modular hardware system which could quickly be installed and applied to different artistically-motivated installation scenarios. As described by Wollscheid on his website: "The “clapper”-system plays objects, rooms or spaces, according to the placement of the electronically driven “clappers” and the specific resonance of the materials and objects to which they are attached... The last version of the system is equipped with a real-time sound-transformation interface - meaning the harmonic spectrum of sound events in spaces or places (voices of engine sounds for example) are recorded, analysed and transformed into percussive patterns."

Wollscheid greatly benefited from the modular design of his clapper-system as it allowed for several site-specific installations to be conducted with different artistic goals in different locations over several years without having to develop completely new hardware systems. As noted by Wollscheid, this system allowed him to create “site- and space-specific compositions which would change with every place and given situation”.

Another example of modular design in music mechatronics can be found in Eric Singer’s ModBots (2001) which are designed with transportability, modularity, and repeated exhibition as primary considerations. Singer’s ModBots are small devices which can be attached to environmental features and which playback compositions with each artefact usually featuring “only one electromechanical actuator (a rotary motor or linear solenoid) which responds to varying degrees of supply voltage regulated by a microcontroller”. Singer noted the advantages enjoyed by the transportable and modular design of the ModBots, “because of their small size, versatile mounting capability and minimal cabling required for installation, ModBots can be configured in limitless arrangements”. Singer noted that the bots have been used to augment “museum stairwells, steel hulled ships and the sculptural work of other artists”.

Modulets (2016) and MalletOTon (2016) are modular mechatronic instruments from the KarmetiK Machine Orchestra. Modulets adopt a similar approach as the ModBots by leveraging multiple centrally controlled individual solenoids or DC motor mechanisms to produce spatialised percussion with visual feedback elements. Alternatively, MalletOTon uses arrays of independent actuator mechanisms whose positions can be adjusted between performances to modify the instrument’s sonic capabilities. Kapur et al. described the benefits of modular design: “a key advantage of modular mechatronic instruments is in their flexibility and ease of deployment”. Kapur et al. concluded that their experience deploying the system in numerous disparate artistic applications, “has emphasized the effectiveness of easily deployed mechatronic instruments”.

These examples of modular musical mechatronic systems from Achim Wollscheid in the 1990s, Eric Singer in the 2000s, and Ajay Kapur in the 2010s demonstrate the advantages of modular approaches to hardware and firmware design within circumstances where the repeated exhibition of similar hardware is expected or desired. As noted by Kapur et al., “while other avenues of mechatronic instrument development focus on the development of increasingly complicated apparatus and interfacing techniques, the utility in a ‘real-world’ environment of a rapidly-installable instrument is undeniable”. These related works have highlighted the benefits of leveraging modular design principles for musical mechatronic projects. Artistically-motivated engineering projects, such as the Acropolis family, often require reevaluation and repositioning of design objectives in reaction to the in-situ performance of the designed systems to realise the project’s artistic objectives. This review of related works has shown how the successful application of modular design principles can facilitate the reuse of firmware and hardware systems to realise distinctly different artistic performances. This beneficial trait for realising Hyper- Soundwalk Series installations has emphasised the importance of following modular design practices for the Explorator genus.

Another valuable takeaway from reviewing these works for the Explorator genus is the successful use of these music mechatronic systems within unusual performance venues. The installation of Singer’s ModBots in stairwells, and Kapur’s Modulets under the seats of audience members during a 2012 performance of Samsara at the REDCAT theatre in Los Angeles, among other examples, provide a foundation for Explorator’s explorations in exhibiting mechatronic instruments in the subjectively more unusual venues of outdoor natural environments.

Outdoor Musical Mechatronics

Many outdoor mechatronic sonic artworks are location-specific and thus installed using permanent materials and infrastructure such as concrete and stone. Trimpin’s large-scale Hydraulis (2005) serves as an example of this type of musical mechatronic installation where hundreds of individual valves produce drips of water that fall to strike resonate pipes to produce different tones. While works such as Hydraulis demonstrate the application of musical mechatronic systems within outdoor environments, to support the pop-up exhibition strategy’s emphasis on ad hoc, temporary installations, this discussion focuses on mobile mechatronic systems. The Logos Man and Machine Orchestra have pioneered mobile mechatronic music design principles with their instruments. Nearly every newly designed Logos instrument features large casters that allow it to be rolled from one venue to another. These casters, along with large lead-acid batteries to power their actuations, allow the instruments to perform in locations without access to mains electricity. Toetkuip (1987), Klankboot (1987), and Le Grand Coucou (1997) are examples of mobile mechatronic instruments created by Raes for Logos with the intent of outdoor exhibition. However, these artefacts are fairly large and often require multiple people to operate. For example, Le Grand Coucou requires a person solely dedicated to steering the device in addition to multiple operators, while Toetkup and Klankboot weigh 180 kg and 240 kg respectively. While the majority of Raes’ outdoor-focused creations are mobile mechatronic instruments, Wolkenrijder (1997), is a notable exception as it is immobile and non-cochlear. Facing skyward, the robot vehicle “rides on clouds” as it collects readings from light sensors that detect the presence and movement of clouds in the sky which it uses to adjust the height of four upward-facing wheels.

Michael Candy is an innovative contemporary mechatronic artist who, like Raes, exhibits work both indoors and outdoors. Candy creates visually striking mechatronic sculptures that leverage movement and light, are biologically-inspired, yet distinctly mechanical. In ETHER ANTENNA (2017), composer Candy collaborated with composer Pauline Anna Strom to create a short film documenting the journey of ten small robotic devices that use sensors and actuators to navigate the environment. They project intelligence and sentience through their careful movements and biologically-inspired aesthetic designs. Instead of fully enclosing and protecting the sensitive electronic components, Candy opts for exposed electro-mechanics, highlighting the fragility of his creations and their mechanical construction. While both ETHER ANTENNA andWolkenrijder are visual works, they show examples of mechatronic devices presented as electronic creatures that are installed in exposed outdoor environments and use environmental data to guide their behaviour.

Raes’s outdoor mechatronic works provide technological and physical design guidance for Explorator by, in part, establishing precedent for lead-acid batteries to power musical mechatronic systems, while simultaneously demonstrating the limitations of this technology due to poor power density. Furthermore, Wolkenrijder ’s use of environmental sensors to drive artefact behaviour in real-time according to cloud-cover provides a reference which the Explorator genus can exploit by leveraging light sensors similar to the Speculator genus. Many permanent musical mechatronic works, like those created by Trimpin, are designed to withstand sustained environmental exposure due to necessity. However, many of the artists that exhibit mechatronic artworks in outdoor environments do not design their devices to explicitly withstand heavy precipitation or other adverse weather events. To account for this vulnerability, artists exhibit their work on days with favourable weather usually as a performance lasting a few hours as the case with Raes work, or to be filmed and digitally distributed as with Candy’s work. However, in the case of Candy, artefacts are installed high up in trees and in locations surrounded by water. These locations, and the kinetic movement of the artefacts resulted in dramatic and zoomorphic displays. Candy’s embrace of mechatronic vulnerability in outdoor environments has impacted Explorator’s design and exhibition by revealing the artistic benefit of showcasing technological vulnerabilities during exhibition to capture and maintain the attention of observers.

This survey of related mechatronic-based artworks has demonstrated several commonalities in their artistic focus, implementation strategy, and technological realisation. Among the key characteristics identified within these works are modular design principles, the visual exposure of electrical systems, device mobility, and the use of environmental sensors to influence artefact behaviour. These practically, logistically, and artistically promising qualities are referenced within the following discussions to synthesise a set of genus-specific objectives to support the Explorator project’s artistic focus and technological realisation.

Mechatronic Music (Writings from my MFA Thesis

Mechatronics is an interdisciplinary field which blends electrical engineering, computer science, control engineering, telecommunications engineering, systems engineering, and mechanical engineering to create a wide variety of electro-mechanical systems from self-driving cars to robotic butlers. The field of mechatronic music is concerned with the musical implications of mechatronic systems within the context of human-robot musical performance. [38], [39] The fields of mechatronics is relevant as many of the installations, and a majority of the performances, introduced in this paper either utilize preexisting mechatronic instruments, are concerned with the creation of new mechatronic instruments, or involve music composed for mechatronic performers. This writing aims to introduce some of the most important individuals and institutions from the field of mechatronic music that have personally impacted or inspired my own work. The remainder of this section introduces Godfried-Willem Raes, Trimpin, and The Machine Lab in chronological order.

Godfried-Willem Raes

Godfried-Willem Raes, born 1952 in Ghent, Belgium, is the mastermind behind the Man and Machine orchestra at the Logos Foundation, founded in 1968, where he proceeds over about 150 new music concerts a year.[40], [41] In 1968/69 Raes co-founded the Logos Ensemble5 while about a year later, in 1970, Raes began performing with the Logos Duo in collaboration with Moniek Darge. In 1973, Raes began acting as programmer for the Philharmonic Society at the Palais des Beaux Arts in Brussels where he held the position until 1988. In 1990, Rae’s commitment to mechatronic instrument creation, and composition, lead to the founding of the Man and Machine Orchestra which currently boasts over forty-five wind, string, percussion, and noise-generation instruments and is one of the oldest robotic orchestras in the world.

Raes’ careful documentation and clean, refined method are an inspiration to how I try to conduct my own research. Raes, his students, and collaborators do a thorough job documenting the process of design and construction of many of the instruments that make up the Man and Machine orchestra. Raes takes meticulous notes on what is accomplished for the given project on each day it is worked on. The foundation freely publishes the information along with pictures of the work in progress on the Logos Foundation’s website. These valuable notes even contain information about what books and websites are used to research various bits of information no matter how minor the detail. These notes serve not only as an invaluable guide to discerning how the individual instrument is structured, but additionally provides insight into the overarching process that Raes and the Logos Foundation employs when creating new mechatronic instruments and performances.

In 1990, with the construction of Autosax, Raes championed the Logos Foundation into a new era, where the foundation shifted its focus away from of the design and use of analog and electronic hybrid sound generating devices and onto the creation of musical robots. Over years performing with purely electronic synthesizers, Raes became convinced that loudspeaker reproductions of sounds are nothing more than a virtualized reality of acoustic sound. He started building new mechatronic instruments and started writing compositions which take advantage of both the extra-human capabilities of the robotics and the liberties afforded onto human instrumentalists when they are able to interact with their instrument in unconventional manners. This has developed into extensive research into the adoption of gesture recognition to interface with the robots of the orchestra. [42]–[44] The instruments that constitute the Man and Machine orchestra are beautiful, unique, and inspirational but it is Raes explorations into the roles of humans 5 The Logos Ensembles was founded in 1968/1969 and has included dozens of members over the decades.

Trimpin

Trimpin, born 1951 in Germany to a father who played wind instruments, is a kinesthetic sculptor, mechatronic inventor, and installation artist based out of Seattle, Washington. Trimpin played wind instruments before developing an allergy to brass after which he began experimenting with modifying electronic devices. In 1980, after studying at the University of Berlin and showcasing installations in Germany, Trimpin moved to America in pursuit of old, used electronic components, which he found difficult to find in Europe. Using repurposed materials, Trimpin currently works creating large-scale sound art in galleries and public spaces throughout the world. [45] Trimpin often employs the tactic of building multiples of a single object that are parallel yet contrasting. This is used to great effect in installations such as Klompen (1990), which features 96 wooden clogs; and Sheng High (2006), shown with Trimpin in Figure 10, which calls upon thirty bamboo pipes to produce more than a two octave range of tones. In both Klompen and Sheng High each one of the similar objects serves to produce a unique note that is vital to realizing the compositions the installation plays. This is a different approach from those taken by artists such as 21

Zimoun (Figure 11), whose installations employ flocks of identical objects who’s emergent complexity is due to sheer numbers and not differences between the objects. For instance, in Trimpin’s installation Sheng High each of the thirty bamboo pipes are tuned to a distinctive pitch. Pattern on the wall created from old, broken CD-ROMs serve as a graphic score which plays back a composition using the pipes. If one of the units is removed, the composition is severely compromised. Trimpin approaches his installations both as a visual artist and as a composer, a philosophy implemented in several of the installations introduced in Chapter 3. [46] Figure 11: Zimoun's 138 prepared dc-motors, cotton balls, cardboard boxes. Just like Godfried-Willem Raes, an idea Trimpin is adamant about is the inherent power of acoustic sound over its loudspeaker counterpoint. Trimpin, along with many of his disciples [47], are adamant that speakers are inadequate at reproducing the experience of the original sound. This sentiment is a tenet in many mechatronic ensembles which tend to only feature mechanically actuated sonic content. Although the work in later chapters often does follow this approach, there are projects, namely Computer Music, which attempt to utilize only the electro-mechanical artifacts of the mechatronic instruments to create the soundscape. Throughout his impactful career, Trimpin has maintained humble and willing to teach the wonders of mechanical art to anyone with interest. This I learned firsthand over the years at CalArts 22 as Trimpin has served as a deeply valued mentor on several of the projects described in the following chapters.

2.4.3 The Machine Lab Figure 12: The Machine Orchestra on stage at the Walt Disney Music Hall. The CalArts Machine Lab is the hub of the Music Technology department as well as the Digital Arts Minor and functions as the technological core of the institute. The Machine Lab is home to the nine mechatronic instruments that make up the Machine Orchestra which consists of a mixed ensemble of human and robotic performers which combines the musical elements of a laptop ensemble with the acoustic affordances of traditional instruments.

Nine mechatronic musical instruments inhabit the Machine Lab at CalArts. BreakBot is a hanging percussion instrument that features a kick drum, a crash cymbal, and a snare6 . Spread throughout the entire room, hidden in the ceiling grid, are twenty Clappers: each consisting of a single solenoid, with a blue LED inside of a ping-pong ball. MalletOTon is a mechatronic marimba that features 48 rotary solenoid actuated rubber headed mallets striking its keys [48]. StringThing is 6 All are actuated with solenoid beaters. The crash symbol has two dampening mechanisms. The snare has a brush, which can be rubbed on the membrane, as well as a beater. 23

made up of three steel strings picked by DC motors with plectrum mechanisms as well as steel post dampener mechanisms activated by solenoids. RattleTron is a percussion instrument that includes an assortment of hand percussion instruments along with three pipes struck with solenoids. MahaDeviBot is an Indian percussion robot that consists of a total of twelve solenoid actuators that strike frame drums, gongs, bells, wood blocks, and finger cymbals. GanaPatiBot is a percussion robot that features five plastic drums of various sizes each with multiple solenoid powered beaters. Lydia is a standup piano with twenty solenoids that strike the strings percussively, sixteen DC motors which ring the strings via custom rubber strikers, and a hacksaw which saws through a large steel bolt at the base of the instrument. JackBox is both a percussion and string instrument which features twelve guitar and bass strings, three cymbals, eleven German beer glasses, an eight key xylophone, and three plastic drums which are all activated using dozens of solenoids7. Tammy consists of six brass bells struck with steel posts and six custom cut wooden keys directly actuated by solenoid plungers. [49]–[53]

By way of my undergraduate and graduate studies with the Music Technology program at CalArts, the Machine Orchestra served as my introduction into the worlds of mechatronic music performance. Over the past four years, I have gotten to know the multifarious instruments in the Machine Lab through hours spent repairing, maintaining, upgrading, and composing for them. My experiences caring for these robots shape the adjunctions made when designing my own mechatronic instruments and installations.

References and Further Readings

[38] N. J. Brown and O. T. Brown, “Mechatronics ‘a graduate perspective,’” Mechatronics, vol. 12, no. 2, pp. 159–167, 2002.

[39] A. Kapur, “A History of Robotic Musical Instruments.,” presented at the International Computer Music Conference, 2005, vol. 31.

[40] L. Maes, G.-W. Raes, and T. Rogers, “The Man and Machine Robot Orchestra at Logos,” Comput. Music J., vol. 35, no. 4, pp. 28–48, 2011.

[41] “Biography Page Godfried-Willem Raes Composer and music maker.” [Online]. Available: http://www.logosfoundation.org/cv-god.html. [Accessed: 25-May-2017].

[42] G.-W. Raes, “NaMuDa: Gesture Recognition for Musical Practice,” Available -Line Www Logosfoundation OrgiiNamuda, vol. 123, 2010.

[43] G.-W. Raes, “Expression control in automated musical instruments,” in a paper presented at the, 2015.

[44] G.-W. Raes, Gesture controlled virtual musical instruments. Ghent, 1999.

[45] Peter Esmonde, Trimpin: The Sound of Invention. 2009.

[46] Trimpin and A. Focke, Trimpin Contraptions for Art and Sound. Seattle [Wash.]: University of Washington Press, 2011.

[47] Jim Murphy, Ajay Kapur, and Dale Carnegie, “Musical Robotics in a Loudspeaker World: Developments in Alternative Approaches to Localization and Spatialization,” Leonardo Music J., vol. 22, pp. 41–48, 2012.

[48] A. Kapur, J. Murphy, M. Darling, E. Heep, B. Lott, and N. Morris, “MalletOTon and the Modulets: Modular and Extensible Musical Robots,” New Interfaces Music. Expr., pp. 69–72, 2016.

[49] A. Kapur et al., “The Machine Orchestra,” in ICMC, 2010.

[50] A. Kapur et al., “The Machine Orchestra: An Ensemble of Human Laptop Performers and Robotic Musical Instruments,” Comput. Music J., vol. 35, no. 4, pp. 49–63, Dec. 2011.

[51] A. Kapur, “DIGITIZING NORTH INDIAN MUSIC,” University of Victoria, 2007.

[52] A. Kapur and M. Darling, “A Pedagogical Paradigm for Musical Robotics.,” presented at the The International Conference on New Interfaces for Musical Expression, 2010, pp. 162–165.

[53] O. Vallis, D. Diakopoulos, J. Hochenbaum, and A. Kapur, “Building on the Foundations of Network Music: Exploring Interaction Contexts and Shared Robotic Instruments,” Organised Sound, vol. 17, no. 01, pp. 62–72, Apr. 2012.

[54] V. Maisonneuve, Everything You Need to Know About Video