Opera of the Future

We are developing a multimodal interface for hand rehabilitation following stroke. EMG forearm sensors read attempted finger presses in disordered limbs, and serve as an input to an expressive feedback interface. Auditory, visual, and tactile cues are presented to support rehabilitation of the representation of finger movements across sensory domains. The multisensory feedback is embedded in a rich task, situated between piano learning and expressive music performance. A user of this system will rehabilitate finger movement while developing an expressive music performance. Imagine a complete shift in the form and function of rehabilitation, towards something empowering, where individuals strive in tandem with tailored interfaces, mapped to push them forward at each step, and as part of fundamentally enriching expressive tasks. Our rehabilitative health care environments can sculpt our minds, while changing our lives, if we invent the right tools.

Early in the opera "Death and the Powers," the main character Simon Powers is subsumed into a technological environment of his own creation. The set comes alive through robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. This environment must assume the behavior and expression of the absent Simon; to distill the essence of this character, we recover performance parameters in real time from physiological sensors, voice, and vision systems. Gesture and performance parameters are then mapped to a visual language that allows the off-stage actor to express emotion and interact with others on stage. To accomplish this, we have developed a suite of innovative analysis, mapping, and rendering software systems. Our approach takes a new direction in augmented performance, employing a non-representational abstraction of a human presence that fully translates a character into an environment.

Many performance artists and interaction designers use human gestures to drive, manipulate, or generate digital media. However, the existing systems for developing mappings between incoming data streams and output media have extremely low-level concepts of “gesture,” forcing the user to focus on the particulars of input sensor or video data, rather than on meaningful and expressive gestures. We are developing a new framework for gestural control of media in performance, allowing users to easily create clear, intuitive, and comprehensible mappings by using high-level descriptions of gestures and of gestural qualities. This system currently is realized in a set of tools for gestural media manipulation in performance and rehearsal, mapping gestural vocabularies and qualities of movement to parameters of interactive visual applications.

Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation, allowing individuals with no musical training to create professional pieces. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.

Our work in Music, Mind, and Health has culminated in a recent PhD thesis, showing the technologies and perspectives required to build on the transformative nature of music to drive specific neurological, physical, and psychological change. A radically new "Personal Instrument" is currently being used by Dan Ellsey, a quadraplegic individual, who controls this interface to sculpt an expressive performance of music in real time. A three-month study of Ellsey's expressive behavior—its potential as well as its limits—resulted in an interface tailored just for him, enabling him to access expressive performance despite his physical disability. This new line of work highlights principles for future instruments and applications, where the impact is in the marriage of the interface and uniqueness of the person. In this way, we are pursuing new design philosophies, technologies, and collaborations within the scientific community, public performance, and clinical research.

Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing one’s deepest thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. This development is based on two guiding principles: first, active music creation yields far more powerful benefits than passive listening; and second, increasing customization of the musical experience is both desirable and possible, as evidenced in our group’s development of Personal Instruments (see Music, Mind, and Health) and Personal Music. Personal Opera goes a step further, using music as the medium for assembling and conveying our own individual legacies, representing a new form of archiving, easy to use and powerful to experience.

Spheres and Splinters is a new work composed by Tod Machover for hypercello, electronics, and responsive visuals commissioned for the Faster than Sound at Aldeburgh Music. The work was premiered with celllist Peter Gregson in the UK in 2010 and had its US premiere as part of FAST Festival of Art, Science, and Technology in celebration of MIT's 150th anniversary. Utilizing audio analysis and a multitude of wireless sensors on the cello and the bow that capture how the instrument is being played, the performer has control over transformations and extensions of the sound produced. This control extends into the ambisonic spatialization of sound in the performance space. The performance data is also used to produce realtime visual accompaniment on an array of LED strips surrounding the cellist.

Toy Symphony combines children, virtuosic soloists, composers, and symphony orchestras around the world to alter radically how children are introduced to music, as well as to redefine the relationship between professional musicians and young people. A complete set of Music Toys will be distributed to children in each host city (including Berlin, Dublin, Glasgow, Manchester/London, and Tokyo), where children will be mentored to create their own sounds and compositions for toys and traditional instruments. A pedagogy for using these Music Toys to teach and to instill a love for musical creativity will also be developed. Final concerts will be presented in each host city including children's compositions and specially commissioned works by young composers, to be performed by children, soloists, and orchestra, playing Music Toys, Hyperinstruments, and traditional instruments.