Last one!

Today I played with spectral flux to change the size of particles across the screen. For each frame of audio a particle is moved from the middle screen. I was surprised how easy spectral flux was and how great it could look.

This link was invaluable.

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

float[] x;
float[] y;

int index;

float[] spectrum;
float[] lastspectrum;

void setup()
{
  size(720, 480, OPENGL);
  hint(DISABLE_OPENGL_2X_SMOOTH);
  background ( 0 ) ;
  smooth();
  noStroke();
  
  x = new float[width];
  y = new float[width];
  
  for (int i = 0; i < x.length ; i++)
    x[i] = (float)i ;
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  fft = new FFT(in.bufferSize(), in.sampleRate());
  fft.noAverages();
  
  spectrum = new float[fft.specSize()];
  lastspectrum = new float[fft.specSize()];
  
  Arrays.fill( lastspectrum, 0 ); 
}

void draw()
{

  fft.forward(in.mix);

  ///------------------------
  // SPECTRAL FLUX
  ///------------------------
  float flux = 0;
  for( int i = 0 ; i < fft.specSize() ; i++)
  {
      spectrum[i] = fft.getBand(i);
      flux += spectrum[i] - lastspectrum[i]; 
      // rectified flux
      // float fluxval = spectrum[i] - lastspectrum[i];
      // flux += (fluxval > 0.0 ) ? fluxval : 0.0;
  }
  System.arraycopy( spectrum, 0, lastspectrum, 0, spectrum.length ); 
  ///------------------------
  
  fill(0,6);
  rect ( 0,0, width, height );
 
  fill(200,60); 
  translate(0,height/2); 
  for ( int i = 0 ; i < x.length ; i++)
    ellipse(x[i],y[i],abs(flux),abs(flux));
  
  y[index++%width] = flux; 
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

Today I played a bit more with the peakiness function but I went back a simple particle system that will evolve over longer periods of time. I borrowed some of the code from yesterday and tried to make it a bit more gradual and I got some interesting results.

I also spent some time today digging into the Java API and found some nice features of Arrays. Sorting is actually quite a nice algorithm and also fill is a great way to just fill up an array with a value. Hooray for not having to write simple things over and over!

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

int numberofparticles = 10000;
float[] x = new float[numberofparticles];
float[] y = new float[numberofparticles];

void setup()
{
  size(720, 480, OPENGL);
  background ( 0 ) ;
  smooth();
  
  
  for (int i = 0; i < x.length ; i++)
    x[i] = (float) i / x.length *  width;
  Arrays.fill(y, height/2);
  
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  fft = new FFT(in.bufferSize(), in.sampleRate());
  fft.noAverages();
}

void draw()
{
  ///------------------------
  // SPECTRAL PEAKINESS
  ///------------------------
  // made this up. not accurate but gives interesting data
  fft.forward(in.mix);
  float sum = fft.getBand(0);
  int peak = 0;
  float centroid = 0;
  
  for( int i = 1 ; i < fft.specSize() ; i++)
  {
    sum += fft.getBand(i);
    centroid += i * fft.getBand(i);
    peak = ( fft.getBand(i) > fft.getBand(peak) ) ? i : peak ;    
  }
  centroid /= sum ;
  float peakiness = (float) peak / centroid;
  ///------------------------

  noStroke();
  fill(0,6);
  rect ( 0,0, width, height );
  
  fill(200,60);
  
  for ( int i = 0 ; i < x.length ; i++)
  {
    ellipse(x[i],y[i],1,1);
    
    y[i] += random(-peakiness,peakiness);
  }
  
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

I am getting quite interested in developing some analysis tools for sound in Java. I am wanting to find the ratio of centre frequency to noise. Today I made a crude algorithm that looks for the peak frequency bin and then uses that in a ratio between spectral centroid. If the peak is off from the centroid then you can assume a distribution that is noisier (aka less pitched). The problems with this algorithm is that it is not normalized to volume and that a centroid of fftsize/2 is noise so if there is a peak at this place then you will get erroneous data. That will take me a little while to find a good solution around this but if was a fun thought experiment.

Today’s patch uses my peakiness measure to draw big bubbles across the screen. More noisy means more crazy drawing. It is actually responding in generally the right way and is kind of interesting to watch. I will definitely be expanding on this idea.

Also today I left in some comments in the code for things that I tried and abandoned. You can see a bit more into my process with these.

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

void setup()
{
  size(720, 480);
  background ( 0 ) ;
  smooth();
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  fft = new FFT(in.bufferSize(), in.sampleRate());
  fft.noAverages();
}

void draw()
{
  ///------------------------
  // SPECTRAL PEAKINESS
  ///------------------------
  // made this up. not accurate but gives interesting data
  fft.forward(in.mix);
  float sum = fft.getBand(0);
  int peak = 0;
  float centroid = 0;
  
  for( int i = 1 ; i < fft.specSize() ; i++)
  {
    sum += fft.getBand(i);
    centroid += i * fft.getBand(i);
    if ( fft.getBand(i) > fft.getBand(peak) )
        peak = i;
  }
  centroid /= sum;
  float peakiness = (float) peak / centroid;
  ///------------------------
  
  noStroke();
  fill(0,6);
  rect ( 0,0, width, height );
  
  translate(width/2, height/2);
  fill(200,60);
  for ( int i = 0 ; i < 10 ; i++)
  {
     float circlesize = random(70,100);
     //ellipse ( random(random(-200,-20), random(50,200)), random(random(-200,-20), random(50,200)) , circlesize, circlesize );
    ellipse ( random(-width/2, width/2), random(random(-200,-20), random(50,200)) * peakiness, circlesize, circlesize );
  }
  
  //filter(BLUR, peakiness);
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

Another day, another experiment. I tried to resurrect my Spectral Flatness code and I think it is a precision error in the geometric mean. I thought about scaling the FFT values but I will have to go back on the math for that to make sure I don’t skew the values the wrong way. For now I have used the arithmetic mean of the spectrum to give me some numbers to play with.

This sketch changes colour at different volumes, currently white for quiet and black for loud, and becomes more animated with noisier spectra (not really but close enough for now). I think I am getting close to something interesting. Maybe once I sleep on it tomorrow will show me the answer.

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

float lastlevel;
float[]  myspectrum;
float[] features;
float spacing;

void setup()
{
  size(720, 480, OPENGL);
  hint( DISABLE_OPENGL_2X_SMOOTH );
  smooth();
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  
  fft = new FFT(in.bufferSize(), in.sampleRate());
  fft.logAverages(50, 2);
  myspectrum = new float[ in.bufferSize() ];

  noFill();
  strokeWeight(8);
  
  spacing = (float) width / fft.avgSize();
  
}

void draw()
{
  background(30);
  
  fft.forward(in.mix);
  float sum = 0;
  for(int i= 0; i < in.bufferSize(); i++)
  {
    myspectrum[i] = fft.getBand(i);
    sum += myspectrum[i];
  }
  
  //==============================
  // SPECTRAL FLATNESS
  // DOESN'T WORK -- geometric mean always evaluates to zero :(
  //==============================
  float flatness = 0;
  double geometricMean = (double) myspectrum[0];
  float arithmeticMean = sum;

  for ( int i = 1; i < myspectrum.length; i++)
  {
        geometricMean *= (double) myspectrum[i];
        if (myspectrum[i] == 0.0)
        {
          println("WARNING: ZERO DETECTED " +frameCount);
          break; // might as well leave since this thing will evaluate to zero
        }
  }

  geometricMean = Math.pow(geometricMean , (1.0f / myspectrum.length) );
  arithmeticMean /=  myspectrum.length;

  flatness = (float)geometricMean / arithmeticMean ;
  
  //==============================
  
  float level = in.mix.level();

  // lowpass filter to smooth out motion
  level += lastlevel;
  level *= 0.5;

  translate( width/2 , height/2 );

  if (10 * log(level) > -50 )
    stroke(0);
  else
    stroke ( 200, 40);

  beginShape();
  for ( int i = 0 ; i < fft.avgSize() ; i++ )
     curveVertex( random(-spacing, spacing) * i, arithmeticMean * random(50,150) - 75 , random (0, -200) );
  endShape();
  
  lastlevel = level;
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

I guess I am at it again. Today I was playing around with an idea for making a spectral centroid visual. I was thinking about FFT displays and I thought that having bars rise and fall isn’t all that visually stimulating, though it is accurate for display. So, what I did was map spectral centroid to the location of a directional light in OPENGL and had it point to a line of boxes whose size modulates with the amplitude of the incoming signal. I also reversed the amplitude mapping so that loud sounds create small boxes. It makes for some nice motion.

I also spent some time looking through the Processing.org bug list in reference to some glitchyness in OPENGL as implemented in Processing. There are varying reports about how to smooth out anti-aliasing and threading issues. It seems the best thing to do is to try out all solutions and see which one works for that particular patch. Although the simplest thing to do to avoid the anti-aliasing problem is to only draw vertical or horizontal lines, but who wants to do that?

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

float r;
float zr;

float lastlevel;
float[]  myspectrum;
float[] features;

void setup()
{
  size(720, 480, OPENGL);

  //http://code.google.com/p/processing/issues/detail?id=217&q=opengl&colspec=Stars%20ID%20Type%20Status%20Priority%20Owner%20Summary
  //http://code.google.com/p/processing/issues/detail?id=53&q=opengl&colspec=Stars%20ID%20Type%20Status%20Priority%20Owner%20Summary
  //noSmooth();
  hint( DISABLE_OPENGL_2X_SMOOTH );
  hint( ENABLE_OPENGL_4X_SMOOTH );
  //smooth();
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  
  fft = new FFT(in.bufferSize(), in.sampleRate());
  myspectrum = new float[ in.bufferSize() ];

  fill(200, 40);
  stroke(30,40);
}

void draw()
{
  background(30);
  lights();

  fft.forward(in.mix);
  for(int i= 0; i < fft.specSize(); i++)
    myspectrum[i] = fft.getBand(i);
  
  //==============================
  // SPECTRAL CENTROID
  //==============================
  float m1 = 0;
  float m0 = 0;

  for (int i = 0; i < myspectrum.length; i++){
    m1 += i * myspectrum[i];
    m0 += myspectrum[i]; 
  }

  float centroid = 0.5;
  if (m0 != 0.0)
  {
    centroid = (m1 / m0) / (float) myspectrum.length ;  
  }
  //==============================
  
  float level = in.mix.level();
  // lowpass filter to smooth out motion
  level += lastlevel;
  level *= 0.5;

  translate( width/2 , height/2 );
  rotateX ( zr++ / 50 );

  // set range of centroid to -1 -> 1
  directionalLight( 200,100,230, centroid * 2 -1 ,0,-1 );
  
  float spacing = (float)width / fft.specSize() ;
  float boxsize = 10;
  for ( int i = 0 ; i < fft.specSize() ; i++ )
  { 
    pushMatrix();
    translate( (i * spacing) - (width/2) + boxsize - 10 ,0,0);
    box(boxsize - 10*log(level) - 30);
    popMatrix();
  }
  
  lastlevel = level;
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

Today’s sketch is a play on the random shape clusters that I seem to be gravitating towards. I used the spectral feature extraction from Day 5 so not all of the code is posted below. You can go to that post and rip the features if you like.

This sketch uses a more coherent model instead of randomness. There are 20 3D boxes in the sketch that have their colour and placement changed based upon the sound coming into the sketch. It is reactive but a little too literal. I think I will try to make some mappings in the future that can take spectral centroids and do something meaningful visually. Sounds like a research project!

import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput in;
FFT fft;

float r;
float zr;

float lastlevel;
float[]  myspectrum;
float[] features;

void setup()
{
  size(720, 480, OPENGL);
  hint( ENABLE_OPENGL_4X_SMOOTH );
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  
  fft = new FFT(in.bufferSize(), in.sampleRate());
  fft.logAverages(50, 2);
  myspectrum = new float[ in.bufferSize() ];
  
  //noFill();
  //stroke(255,40);
  fill(200, 40);
}

void draw()
{
  background(30);
  
  fft.forward(in.mix);
  for(int i= 0; i < in.bufferSize(); i++)
    myspectrum[i] = fft.getBand(i);
  
  features = drawSpectral(myspectrum);
  
  fill(features[0] * 200 + features[1] * 200 + 100, 40);
  
  float level = in.mix.level();

  // lowpass filter to smooth out motion
  level += lastlevel;
  level *= 0.5;
  
  if ( 10 * log (level) > -50 )
     zr++;

  translate( width/2 , height/2 );
  rotateZ ( zr / 50 );

  for ( int i = 0 ; i < 20 ; i++ )
  {
    rotateY( r / 300 );
    rotateX( r / 500 );
    translate(log(level)*i,0,0); // comment this line out for more standard shape
    box(width/4 + 100 + ( 10 * log(level)) +i*3);
  }
  
  r++;
  lastlevel = level;
}

void stop()
{
  in.close();
  minim.stop();
  super.stop();
}

void mouseClicked()
{
   saveFrame(); 
}

View this code on GitHub

float[] drawSpectral(float[] spec)
{
  
  //==============================
  // SPECTRAL CENTROID
  //==============================
  float m1 = 0;
  float m0 = 0;

  for (int i = 0; i < spec.length; i++){
    m1 += i * spec[i];
    m0 += spec[i]; 
  }

  float centroid = 0.5;
  if (m0 != 0.0)
  {
    centroid = (m1 / m0) / (float) spec.length ;  
  }

  //==============================
  // SPECTRAL ROLLOFF
  //==============================
  float perc = 0.8;
  float[] sumWindow = new float [ spec.length ];
  float total = 0;	
  float sum = 0;
  float specRolloff = 0;
  boolean specdone = false;
  for( int i = 0; i< spec.length; i++)
  {
    sum += spec[i];
    sumWindow[i] = sum;
  }
  total = sumWindow[spec.length-1];

  for ( int i = spec.length-1; i > 1 ; i--){
    if (sumWindow[i] < 0.8 * total)
    {
      specRolloff = (float) i;
      specdone = true;
      break;
    }
  }
  if ( !specdone )
    specRolloff = float(spec.length - 1);
  specRolloff /= spec.length;

  //==============================
  // SPECTRAL COMPATCTNESS
  // code snippet from jAUDIO  
  //==============================
  float[] mag_spec = new float[ spec.length ];
  for (int i = 0; i < spec.length; i++)
  {
    mag_spec[i] = abs( spec[i] );
  }

  double compactness = 0.0;
  for (int i = 1; i < mag_spec.length - 1; i++) {
    if ((mag_spec[i - 1] > 0.0) && (mag_spec[i] > 0.0) && (mag_spec[i + 1] > 0.0)) 
    {
      compactness += Math .abs(
          20.0 * Math.log(mag_spec[i])
          - 20.0 * (Math.log(mag_spec[i - 1]) 
            + Math.log(mag_spec[i]) 
            + Math .log(mag_spec[i + 1])) 
          / 3.0);
    }
  }

 float[] f =  { centroid, specRolloff, (float)compactness };
 return f;
}

View this code on GitHub

Got a little busy today so I made a quick sketch to see if I understood making shapes with OPENGL. I do!

import javax.media.opengl.*;
import processing.opengl.*;


int jf;

void setup(){
  size(720,480, OPENGL);
  hint(ENABLE_OPENGL_4X_SMOOTH);
  fill(240,30);
}


void draw(){

  background(0);
  jf = int(random(10,50));
  
  beginShape(TRIANGLE_FAN);
  for (int i = 0; i < jf; i++)
    vertex( random(0,width), random(0, height), random(0,100));
  endShape(); 

}

void mouseClicked(){
  saveFrame();
}

View this code on GitHub

BoxySean sent me a link to SuperDraw yesterday and I was really blown away. Then I looked at who was responsible and it made a lot of sense. So today I thought I would try and do some 3D drawing using sound levels. I originally tried to make some shapes and have them connect and then I realized I would have to come up with a smart system for stacking shapes. That might be another day. For now I have a little reactive audio drawing tool that I can expand on in the future.

import ddf.minim.*;

  Minim minim;
  AudioInput in;

int r;
ArrayList shapes = new ArrayList();

void setup()
{
  size(720,480, P3D);
  hint(ENABLE_OPENGL_4X_SMOOTH);
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);

  shapes.add( new PVector(10,20,30) );
  stroke(240,50);
  fill(180,100);
}

void draw()
{
  background(0);
  
  shapes.add ( new PVector ( in.mix.level()*width , in.mix.level()*height, 10*log(in.mix.level())  ) );
  
  translate(width/2, height/2);
  rotateY((float)r++/300); 
  beginShape();
  for ( int i = 0 ; i < shapes.size() ; i++ )
  {
     float[] p1 = ( (PVector)shapes.get(i) ).array();
     vertex( p1[0], p1[1] ,p1[2]  );  
  }
  endShape();
  
}

void mousePressed()
{
  saveFrame();
}

View this code on GitHub

Today I had an idea to see if I could kaleidoscope an image or make a mirror. This was something I was thinking of as a way to illustrate mono signals and then I could modify the images as the signals diverged. I played around with rotating a bit and then remembered that it could get ugly and convoluted so I did a bit of googling and the Processing forums showed a gem: scale(). I can set the scale to negative values in either the x or y dimension (or z if you want to get crazy) and then move the image accordingly and all is fine.

I set up minim to blend a noise image into my scene if there is an onset and then I played around with rotating boxes again. Look, it isn’t black and white!

import ddf.minim.*;

Minim minim;
AudioInput in;

PImage img;
PGraphics graph;
int r;
void setup() 
{
  size(720,480,P2D);
  
  minim = new Minim(this);
  in = minim.getLineIn(Minim.STEREO, 512);
  
  img = createImage(width,height, ARGB);
  graph = createGraphics(width/2, height, P2D);
  graph.smooth();

  graph.beginDraw();
  graph.background( color (290, 39, 111, 80));
  graph.stroke(200);
  for (int i = 0 ; i < 30 ; i++)
    graph.line(0,0,random(graph.width/2, graph.width) , random(graph.height/2, graph.height));
  graph.endDraw(); 

  img.loadPixels();
  for (int i = 0 ; i < img.pixels.length ; i++)
     img.pixels[i] = color( sin(random (0,TWO_PI)) *122 + 122);
  img.updatePixels();  
}


void draw()
{
  background(sin((float)frameCount/20) * 20 + 20 );
  image (graph,0,0);
  
  graph.beginDraw();
  graph.stroke(200);
  graph.rotate(r++);
  graph.rect(graph.width/2, graph.height/2, 40,40);
  graph.endDraw(); 
  

  pushMatrix();   
  scale(-1.0, 1.0);

  image(graph,-width,0);
  popMatrix();
    
  if ( in.mix.level() > 0.1 )
  {
     blend(img, 0,0,width,height, 0,0, width,height, DODGE);
     saveFrame();
   }
}

void mouseClicked ()
{
 saveFrame();
}

void stop()
{
  in.close();
  minim.stop();  
  super.stop();
}

View this code on GitHub

No sound today. I was thinking of making something that might look like a Shepard’s Tone and I came up with this simple idea of a complex polygon that expands outwards into the viewer. I got it going and then I thought it would be cool for it to be endless. I will have to think how I can do that practically. I have a feeling it will borrow some waterfall code and test to see if vertices are out of visual range and then put them back into the pile. I think I will need to sleep on that one.

import javax.media.opengl.*;
import processing.opengl.*;

int r;
int numberoflines = 9000;

float[] x = new float [numberoflines];
float[] y = new float [numberoflines];
float[] z = new float [numberoflines];

void setup(){
  size(720,480, OPENGL);
  hint(ENABLE_OPENGL_4X_SMOOTH);
  smooth();
  fill(240,30);
  stroke(140,40); // try 240,30
  
  for (int i = 0 ; i < numberoflines ; i++)
  {
    x[i] = random(-width/2, width/2);
    y[i] = random(-height/2, height/2);
    z[i] = random(0,500);
  }
}


void draw(){

  background(0);

  translate(width/2, height/2,-500);
  rotate((float)r++/800);
  //rotateY((float)r++/500); // comment out this line for better motion
  noFill();
  strokeWeight(4);
  beginShape();
  for (int i = 0; i < x.length; i++)
  {
    vertex( x[i], y[i], z[i]);
  }
  endShape(); 
  
  for (int i = 0; i < x.length; i++)
  {
    x[i] += random(-1,1);
    y[i] += random(-1,1);
    z[i] += random(-1,1);
    
    z[i] += random( (float) i * .001);
  }  
}

void mouseClicked(){
  saveFrame(); 
}

View this code on GitHub