Sound Sculpture Playground

I am working on an interactive sonic playground that allows seeing and non-seeing individuals to create - and optionally to collaborate with others on creating - sonic compositions in real time by setting kinetic sculptures into motion either by touching or wearing the sonic objects on their bodies. I began this ongoing project last year at NYU's ITP (Interactive Telecommunications Program) last year and I will work on it again this year.

Last year I learned how to take sensor data from accelerometers and and other motion sensors and feed it into MaxMSP to create real-time generative sound compositions that incorporate sensor data into their algorithm and allow it to impact how the instrument will sound in the future. These ever-changing and impressionable compositions live inside the objects on a raspberry pi and connect to speakers/headphones via bluetooth. Snippets of surrounding space is sometimes captured and fed into the algorithm also, making the space sound part of the permanent DNA of the objects and surfacing again in future compositions. In this way, the sonic objects aren't just instruments but are entities that change over time along with us and affected by us.

So far I have simple prototypes that work and I now have a second year at ITP to create a whole collection of these "instruments", each one with it's own "character" - it's own behaviors and sonic qualities. I intend to use a variety of materials and textures that are interesting to touch and that are durable - plush, inflatable, conducive sting fibers, resin, resin foam, 3d-printed with PLA, and even cement.

The collective soundscapes that result from people engaging with the objects and sometimes with each other are portraits of those interactions and spaces and a testament to collaboration, creativity, and play that are so important for human development and self reflection. These qualities separate us from other animals. This sort of tactile, sonic, mindful making in a space that is highly abstracted from familiarity and removed from the bustle of everyday life is also intended to be a place of meditation that the modern human so desperately needs.

video Block
Double-click here to add a video by URL or embed code. Learn more
 An Egg and a Banana but Abstract (Sound Object Series) at ITP Spring Show 2018.

An Egg and a Banana but Abstract (Sound Object Series) at ITP Spring Show 2018.

 Some recent additions

Some recent additions

 First iteration. A rolly Poly pair.

First iteration. A rolly Poly pair.

The sound playground is a group of objects that emit sound when set into motion, whose sonic behavior changes over time, and whose interactions cannot be predicted with certainty. It encourages play, exploration, and the relinquishing of complete control in the creative process.

The sound playground is a group of kinetic sculptures whose sonic behavior changes over time and whose interactions cannot be predicted with certainty. It encourages play, exploration, calm, and the relinquishing of complete control in the creative process. Fascinated with human ability to remain playful throughout the life cycle, I aim to build a sonic playground that allows people of all ages to create subtle, ever-changing, and sometimes surprising, sonic compositions by setting kinetic sculptures into motion.
In addition to play, I am examining the human interactions with things that they cannot control or learn as well as change over time. The objects absorb bits of the surrounding noise and insert these bits into the existing algorithmic composition, somewhat like new snippets of DNA.  An object's program also changes in response to how it was moved. The experiences of an object is incorporated into it's future behavior, but it changes alongside us using it's own logic. The interactions invite users to examine what it means to them to be in control, especially in control of the creative process. Can we accept this degree of independence and be at ease with it?
 

Parts of the project that are done: 

For the spring show I created objects that emit sound and record sound that changes the composition over time. This is a good start.  See documentation images and video at the top of the page. 

 

To Do: 

1. Use rhino to model the notched shape or create it out of styrofoam?  

2. Use find out a good stopper to close all of the enclosures, both the existing paper mache ones.

3. Finish writing python program

4. set up a way to modulate synth sound without a max

5. Finalize whether i want to use raspberry-pi and go wireless? 

  

Accelerometer 

Above is an early prototype of an accelerometer modulating audio. 

 

Enclosure shapes

I decided on the 6 shapes, hoping to make 3 by the Spring Show.

 

1. Roly Polly with a marble inside to lengthen the duration of the movement. A marble inside inside will  lengthen the duration of the movement..

Going for a Brancucci meets Ernest Neto vibe here. First one is Ernesto:

 Another earlier prototype of rolly poly behavior made of a baloon, tape, plaster, and paint.

Another earlier prototype of rolly poly behavior made of a baloon, tape, plaster, and paint.

 Sliced off the top, to be filled with speaker, accelerometer, raspberry pi. 

Sliced off the top, to be filled with speaker, accelerometer, raspberry pi. 

 Japanese handsaw at the show in ITP did the trick. Next step is using a hard cardboard or wooden plank to create a circular closure with slots that will help let our sound. 

Japanese handsaw at the show in ITP did the trick. Next step is using a hard cardboard or wooden plank to create a circular closure with slots that will help let our sound. 

 a leaner! :) kind of nice how it can lean against a wall. 

a leaner! :) kind of nice how it can lean against a wall. 

Large GIF (422x590).gif
Large GIF (422x590).gif
 

2. a notched, agg-crate enspired boat-like shape that rocks back and fourth. A marble inside will  lengthen the duration of the movement. Right images  inspiration by Amanda Martinez, curretnly on view at Victori and Mo. She uses hand carved wood and plaster and sometimes casts these assembled shapes in resin. Her sculptures are solid but I want mine to be hollow or at least have an opening big enough for electronics and also to be closable.

IMG_5079.JPG
Screen Shot 2018-04-11 at 4.04.42 PM.png
 

3. a shape with facets that can play different qualities of sound when placed on each facet.

 
 

Future shapes

 

4. a pendulum (also maybe with a marble of some magnetic disruptor?)

 

5. If possible, a shape pair that has visual indicators (matching color flat surfaces) that the pair needs to be together. The two units will have proximity sensors (RFduinos) that will modulate the sound when they are moved closer together or. are moved apart.

IMG_4899.JPG

far future

6. plush shapes and plush sound suitsinspiredby Ernesto Neto’s Humanoids. 

FullSizeRender.jpg
IMG_0752.JPG

7. One shape that has a fabric, soft aspect, like a frond-like tassel or mohawk-like tuft.  The last three images are by the artist duo Chiaozza.

Below is the first sculpture of the group. 

IMG_0543.JPG
IMG_0542.JPG
IMG_0574.JPG
 

 

7. Shapes that are filled with sand or marked and make a sound when you turn them over.  

 

FullSizeRender.jpg

 

Color

Some color ideas are below. As you can see in the final prototype, I ended up not going with these palettes this round.

 Currently I am creating paper mache prototypes and experimenting with color.

Currently I am creating paper mache prototypes and experimenting with color.

inspirations

 
 

Electronics

List  of things to have:

  • 3 Arduinos, or Rasp pi?
  • 3 Accel,
  • learn Rhino and make or download the lip of a jar or screw cap to 3d print and then build on to of them with paper mache. 
  • extra battery
  • 3 speakers 

The goal of this project is to explore how to modulate synthesized sound emitted from objects set off into motion. At one point I added a layer of complexity - to change the sound not only based on the movements but also based on the proximity of objects. Due to the complexity of the task, I was advised to scale down the project and leave out the interaction between the objects themselves and focus on the user-object interaction for the first step.

The objects are, currently 3-d printed and paper mache, hollow enclosures of various oragnic and geometric shapes with a weighted bottom. Each one is equipped with an Arduino Uno, Accelerometer, and a HC-05 bluetooth friend used to communicate acceleration of the objects to Max MSP, a software that generates sound, on my laptop. 

 

Sounds

Using Fmpeg and a python script I am finding ways to insert choped up sounds into existing chunks so that the composition of one of the shapes is always evolving and taking abstracted input from the environment. 

Sculpture 1:

Abstract organic mashup of environment over ambient noise

Example of very rough current file. Help from Roland Arnold and Leon Eckert. 

Some sounds similar to these in my Sound and Pattern series project

Possibly also these ambient string instruments that are generating using an algorithm, which also is altered when movement of the sculpture is detected by the accelerometer.

 

Sculpture 2:

Percussion - different rhythms on different sides/planes of the multifaceted enclosure.

 

Sculpture 3:

 Synth or low drone that is modulated with movement of the rolly poly shape

Possibly also these ambient string instruments that are generating using an algorithm, which also is altered when movement of the sculpture is detected by the accelerometer.

 
 

Functionality

Problem & Solution 1

Another simplification I made to the project is I took out the bluetooth speakers from the objects. The reason bluetooth speakers posed a problem was that Max MSP is a program on the computer and the computer can only have one bluetooth connection at a time.  

Problem & Solution 2

Originally I wanted to use an RFID controller, and two RFID transponder stickers to measure the proximity of the objects to one another but I could not find adequate documentation even with the help of a physical computing pro resident, Aaron Parsekian (he was very helpful on other aspects of the project). Furthermore, I was advised that the transponders would require extremely close proximity to one another, like a key card reader, which posed a problem for my project. I am interested in exploring how objects can change sound on much farther distances - somewhere between 1 foot and 5 feet ideally. For that I was advised, by my instructor Ayodamola Okunseinde and by Tom Igoe (professor at ITP and cofounder of Arduino!)  to use an RFduino - a radio frequency Arduino that could be programmed to work within that range. I decided to leave this for my future projects.

I was grateful and delighted to work with Aaron Montoya-Moraga on creating wireless communication between the Arduino and Max MSP. He is a sound artist and resident researcher here at ITP who has extensive experience with Max MSP (I have taken several of his workshops in Max MSP) and who has an engineering background. Our documentation for this project is here

 Sketches of shapes

Sketches of shapes

Potential ways to showcase the sound objects:

In the first iteration, I had a pair, represented below. Because the sound emits from the laptop it is possible to use headphones in a crowded exhibition, like the ITP show, for example. 

IMG_9299.JPG

Another way for the sculpture to live is to sit on a bookshelf and emit relaxing drone sounds upon activation.

 

Fabrication

The first iteration was 3d printed. I made a number of paper-mache ones recently and I plan to use styrofoam and paint to create some of the more complex sctructures. 

IMG_8587.JPG

I decided to make a 3-D model of an irregular oblong shape, reminiscent of an egg, using Rhino software.  Chester Dols helped me create these shapes in Rhino and showed me how to use the 3d printer. He's super helpful and I admire his work. 

First, I printed the enclosure at 20% to see if it is structurally sound:

IMG_8593.JPG

I then printed it full size (about 5 inches in diameter) and had one succesful print, fillowrd by 4 failed attempts. The only printer that would print these is the .6 nozzle, which was then never free again. I have yet to schedule it and attempt to print the rest of the enclosures full size. Alternatively, I may try to print smaller enclosures in the .6 nozzle Ultimakers. 

IMG_8606.JPG

Another variation of these shapes is this:

 

The components

Large GIF (416x228).gif

I'm using two accelerometers and RFID controllers to modulate the sound that eminates from the two objects. 

Sound

Option 1

I am using Max/MXP to create synth patches, which will be modulated by the serial input from the two accelerometers (via Arduino 101, which has a wifi shield) in the sculptures. The sculptures are roly-poly toy shapes that wobble when nudged - this is what creates the accelerometer values. Max will send the sound output via blue tooth to a bluetooth speakers that rest inside the sculptures themselves. Concern: I am a little worried there ill be lag when using wifi and bluetooth to send data back and fourth from the computer. 

version_1_w_Max.jpg

 

Option 2

Another option that might produce immediate sound upon touch would be to use micro SD cards in an Arduino Uno. I am not sure if i could use the serial values of an accelerometer to modulate pre-recorded sounds. 

Another concern, is that the bluetooth speakers aren't loud enough. I will test this out. May need to get an amplifier. 

version_2_w_sdcard.jpg

 

 

Code for Accelerometer 1: MMA_7455


 

Code for Accelerometer 2 - ADXL335 - 5V