Pattern and Sound Series - P5js and Tone.js

Using p5js and Tone.js, I created a series of compositions that reconfigure temporarily on input from Kinect, a motion sensor, and generate soothing and energizing synthetic sounds that match the visual elements of the piece. The people in front of the projected visuals and sensor, in effect, create sonic compositions simply by moving, almost like a conductor in front of an orchestra but with more leeway for passive (if one simply walks by) or intentional  interaction.

The sound is easy to ignore or pay attention to. It is ambient and subtle. 

The next steps are to create more nuanced layers of interactivity that allow for more variation of manipulation of the sound and the visuals. Right now I am envisioning the piece becoming a subtle sound orchestra that one can conduct with various hand movements and locations of the screen. 

Composition 1: Ode to Leon's OCD

In the editor.

Composition 2: Allison's Use of Vim Editor

Mouseover version for testing on the computer.

In the editor using Kinect.

Composition 4. "Brother"

In the p5js editor.

For this synchopated piece I sampled sounds of tongue cliks and and claps found on No tone.js was used in this piece.  

Composition 5. "Free Food (in the Zen Garden)"

For this composition I used brown noise from Tone.js. 

In the p5js editor.

After using mouseOver for prototyping, I switched over to Kinect and placed the composition on a big screen to have the user control the pattern with the movement of the right hand instead of the mouse. 



I'm uising Tone.js, written by Yotam Man.


P5js and Adobe Illustrator.

Final Compositions



Initial Mockups


Sketches of ideas



The simple grid-like designs I am creating are inspired by Agnes Martin's paintings. 

Minimalist and sometimes intricate but always geometric and symmetrical, her work has been described as serene and meditative. She believed that abstract designs can illicit abstract emotions from the viewer, like happiness, love, freedom. 



Composition 1: "Ode to Leon's OCD" using mouseover code.

Composition 2: "Allison's Use of Vim Editor" with Kinectron code, using Shawn Van Every and Lisa Jamhoury's Kinectron app and adapting code from one of Aaron Montoya Moraga's workshops.

Composition 2 with mouseover for testing on computer.

Composition 3: "Isobel" with mouseover.

Composition 4 "Brother" with mouseover.

Composition 5. "Free Food (in the Zen Garden)"

Ram Bearer in P5JS (with sound)

I had mocked up this ram-bearer and ram combo quickly in Photoshop using the timeline function of Photoshop just to get a feeling for placement. I then created the above animation above using the PNG files with transparent backgrounds and the translate function.

Here is the gif animation I had created initially:

The sketch for the idea came out from an animation timeline i created for Animation class:



Why buttons

Initially I wanted to drag and drop the images but I had trouble with the code for that one so opted to make buttons. My thought was to make them transparent and just have the user click the heads to move them, but I realized I prefer the web2.0 aesthetic of the buttons floating right over the image. It breaks the seriousness of the emotions that could be evoked by the sculpture (uneven power dynamics, helplessness, etc.) and creates a feeling of agency and ease, as if to say "Wouldn't it be nice if changing power dynamics could be as easy as pressing a button!"



Another use for the buttons is the sound I tied to them. For the lamb button I used the Rhinoceros sound and for the man button I used the Screaming Alien sound. Both are from



var imgBase; // Declare variable 'imgBase'.
var imgLambHeadUpright;
var imgLambDownSleeping;
var imgLambDownOpenEyes;
var imgLambLookingRight;
var imgLambLookingAtMan;
var imgBlank;
var imgHeadMan;
var i;
var lambState = [];
var x;
var y;
var xLamb;
var yLamb;
//var buttonMan;
//var buttonLamb;
var rhino;

function setup() {
  createCanvas(540, 730);
    rhino = loadSound("350424__benjaminharveydesign__rhinoceros-trumpeting-angry.wav");
    blabber = loadSound("188592__alex-audio__screamalien-norberd (1).wav")
  xLamb= 0;
  y=height / 8;
  yLamb= height / 8;
  imgBase = loadImage("bodies_noheads_base.png"); // Load the image
  imgLambHeadUpright = loadImage("lamb_head_upright.png");
  imgLambDownSleeping = loadImage("lamb_head_sleeping.png");
  imgLambDownOpenEyes = loadImage("lamb_head_open_eyes.png");
  imgLambLookingRight = loadImage("lamb_head_looking_to_right_softer.png");
  imgLambLookingAtMan = loadImage("lamb_head_looking_at_man.png");
  imgBlank = loadImage("blank.png");
  imgHeadMan = loadImage("head_man.png");

  var i;

  lambState[i] = [imgLambHeadUpright, imgLambHeadUpright, imgLambDownOpenEyes, imgLambLookingRight,
    imgLambLookingAtMan, imgBlank];

  buttonLamb = createButton('Lamb');
  buttonLamb.position(420, 220, 4, 4);

   buttonMan = createButton('Man');
  buttonMan.position(180, 230, 4, 4);


function nextLambState() {

 //moving on the horizontal axis
  xLamb = xLamb -30;
  // Moving up at a constant speed
  yLamb = yLamb -1;
  if (rhino.isPlaying() == false) {

function nextManState() {
  // moving on the horizontal axis
  x = x -30;
  // Moving up at a constant speed
  y = y - 14;
  if (blabber.isPlaying() == false) {

function draw() {
 background(10, 245, 95);
// background(245);
//   background(224, 254, 75);

  // Displays the image at its actual size at point (0,0)
  //image(imgBase, 0, 0);
  // Displays the image at point (0, height/2) at half size
  image(imgBase, 0, height / 8, imgBase.width / 1.7, imgBase.height / 1.7);
  image(imgLambLookingRight, xLamb, yLamb, imgLambLookingRight.width / 1.7, imgLambLookingRight.height / 1.7);
  image(imgHeadMan, x, y, imgHeadMan.width / 1.7, imgHeadMan.height / 1.7);

  //invisible button for man head options
  // noStroke();
 // fill(255, 255, 255, 0);
 // ellipse(260, 200, 160, 160);

  //invisible button for lamb head options

  // noStroke();
  //fill(255, 255, 255, 0);
  //ellipse(420, 220, 160, 160);

Struggles and Potential Changes

Struggle  1

I like how this experience turned out but would like to, in addition, if time allows, explore my initial idea. 

Initially I wanted to create buttons that switched the lamb and man head positions (there are several PNGs, some with the head positioned a certain way, some with eyes closed). I think it would be fun to make a mix and match situation. Perhaps the user wants the sculpture to have no heads at all, or maybe they fancy that both heads are of the lamb, or both of the man. I tried using the display and hide functionsbut to no avail. Will look into it for future projects. 

I also tried using arrays in order to change the state of the elements on mouse clicks, but i could not conjure up the images when I called on these arrays. The only way I managed to have images show up was using the image command, which is not dynamic. I tried printing to see if the array is even "taking" the image I assigned to it but it looks like maybe the array is empty:



Here are some of the permutations, some with eyes open and some closed that I would have liked to appear for the mix-and-match version of this work:


Struggle 2.

I wanted to add psychedelic colorful strips to the back of the png images but when I did, they made the png images disappear. I could add the boxes on top, however, which looked cool but not what i wanted. 

Screen Shot 2017-11-13 at 8.00.10 AM.png


Inspiration and sources

I used a photograph of the greek Kriophoros - Lamb Bearer - statue. There are many similar ones across cultures but this one appealed to me because it's the most classic for the Western world and I feel more comfortable defacing it. This is probably why Terry Gilliam use classic statues from antiquity also. Terry Gilliam was the inspiration here - the slapstick, irreverent, over the top aesthetic is borrowed from him. 

It's possible that Jodorowsky's themes of the slave becoming the master, or the son killing the father to take his place, have also played a role. Both Jodorowky and Gilliam are pretty playful and this is why I like their work.