Vector Rails
Clinker (Electronics) + Adam Tindale (Drums)
http://github.com/drart/Temporal-Extinction-Event
The above link will take you to the sourcecode for the visual engine for Temporal Extinction Event. This software was written entirely in Processing using insipirations from the Creative Pact 2010 code sprint from September.
The video below was shot by Kari McQueen and edited by Clinker. Enjoy!
This work was generously supported by the Alberta Foundation for the Arts.
Hi Everyone,
Though I have been quiet I have been busy. I really like doing sprint work, like Creative Pact 2010 in September, so I have decided to make my own theme: Documentation. This week I am going to write a new post every day with documentation of what I have been doing in the past few months. I have been playing with software and hardware and doing some shows. You will get a little taste of that in different forms every day this week. I hope you find something that you like!
A great tutorial on OffScreen graphics rendering on blog.blprnt.com made me want to release some code today. I have been working on some offscreen graphics rendering to create masking effects for an upcoming performance. Processing offers PImage and PGraphics as objects that hold raw pixels or vector graphics. These are kept somewhere in memory and not drawn to the screen unless you put them there using image(). This is much faster than drawing to the screen in almost every instance.
Using the blend() command with the MULTIPLY option you can apply a masking effect to your sketch. For the example below the mask is a PGraphics object that contains a line which scrolls down the screen. Since the background is set to black (or 0) when it is multiplied against the image drawn on the screen it becomes black (because anything times 0 is 0). If this example is working properly you should see a red line move down the screen. This is because the image drawn to the screen is just a red background.
PGraphics g;
int x;
void setup()
{
size( 720, 480);
// create the mask
g = createGraphics(width,height, P2D);
}
void draw()
{
background(244,90,10);
// draw the mask
g.beginDraw();
g.background(0);
g.stroke(255);
g.line(0, x%height, g.width, x++%height);
g.endDraw();
// apply the mask to the screen
blend(g,0,0, width,height, 0,0,width,height,MULTIPLY);
}If you’d like to get fancy you can draw your image and mask offscreen and then send the result to the screen. Here is a little code snippet to get you started on your journey.
PImage img;
Pgraphics msk;
//.... do some drawing in both
// apply the mask to the image, both offscreen
img.blend(msk, 0,0,img.height, img.width, 0,0,img.width,img.height,MULTIPLY);
// draw the masked image to the screen
image(img,0,0,width,height);Last one!
Today I played with spectral flux to change the size of particles across the screen. For each frame of audio a particle is moved from the middle screen. I was surprised how easy spectral flux was and how great it could look.
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
float[] x;
float[] y;
int index;
float[] spectrum;
float[] lastspectrum;
void setup()
{
size(720, 480, OPENGL);
hint(DISABLE_OPENGL_2X_SMOOTH);
background ( 0 ) ;
smooth();
noStroke();
x = new float[width];
y = new float[width];
for (int i = 0; i < x.length ; i++)
x[i] = (float)i ;
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
fft.noAverages();
spectrum = new float[fft.specSize()];
lastspectrum = new float[fft.specSize()];
Arrays.fill( lastspectrum, 0 );
}
void draw()
{
fft.forward(in.mix);
///------------------------
// SPECTRAL FLUX
///------------------------
float flux = 0;
for( int i = 0 ; i < fft.specSize() ; i++)
{
spectrum[i] = fft.getBand(i);
flux += spectrum[i] - lastspectrum[i];
// rectified flux
// float fluxval = spectrum[i] - lastspectrum[i];
// flux += (fluxval > 0.0 ) ? fluxval : 0.0;
}
System.arraycopy( spectrum, 0, lastspectrum, 0, spectrum.length );
///------------------------
fill(0,6);
rect ( 0,0, width, height );
fill(200,60);
translate(0,height/2);
for ( int i = 0 ; i < x.length ; i++)
ellipse(x[i],y[i],abs(flux),abs(flux));
y[index++%width] = flux;
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}Today I played a bit more with the peakiness function but I went back a simple particle system that will evolve over longer periods of time. I borrowed some of the code from yesterday and tried to make it a bit more gradual and I got some interesting results.
I also spent some time today digging into the Java API and found some nice features of Arrays. Sorting is actually quite a nice algorithm and also fill is a great way to just fill up an array with a value. Hooray for not having to write simple things over and over!
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
int numberofparticles = 10000;
float[] x = new float[numberofparticles];
float[] y = new float[numberofparticles];
void setup()
{
size(720, 480, OPENGL);
background ( 0 ) ;
smooth();
for (int i = 0; i < x.length ; i++)
x[i] = (float) i / x.length * width;
Arrays.fill(y, height/2);
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
fft.noAverages();
}
void draw()
{
///------------------------
// SPECTRAL PEAKINESS
///------------------------
// made this up. not accurate but gives interesting data
fft.forward(in.mix);
float sum = fft.getBand(0);
int peak = 0;
float centroid = 0;
for( int i = 1 ; i < fft.specSize() ; i++)
{
sum += fft.getBand(i);
centroid += i * fft.getBand(i);
peak = ( fft.getBand(i) > fft.getBand(peak) ) ? i : peak ;
}
centroid /= sum ;
float peakiness = (float) peak / centroid;
///------------------------
noStroke();
fill(0,6);
rect ( 0,0, width, height );
fill(200,60);
for ( int i = 0 ; i < x.length ; i++)
{
ellipse(x[i],y[i],1,1);
y[i] += random(-peakiness,peakiness);
}
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}I am getting quite interested in developing some analysis tools for sound in Java. I am wanting to find the ratio of centre frequency to noise. Today I made a crude algorithm that looks for the peak frequency bin and then uses that in a ratio between spectral centroid. If the peak is off from the centroid then you can assume a distribution that is noisier (aka less pitched). The problems with this algorithm is that it is not normalized to volume and that a centroid of fftsize/2 is noise so if there is a peak at this place then you will get erroneous data. That will take me a little while to find a good solution around this but if was a fun thought experiment.
Today’s patch uses my peakiness measure to draw big bubbles across the screen. More noisy means more crazy drawing. It is actually responding in generally the right way and is kind of interesting to watch. I will definitely be expanding on this idea.
Also today I left in some comments in the code for things that I tried and abandoned. You can see a bit more into my process with these.
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
void setup()
{
size(720, 480);
background ( 0 ) ;
smooth();
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
fft.noAverages();
}
void draw()
{
///------------------------
// SPECTRAL PEAKINESS
///------------------------
// made this up. not accurate but gives interesting data
fft.forward(in.mix);
float sum = fft.getBand(0);
int peak = 0;
float centroid = 0;
for( int i = 1 ; i < fft.specSize() ; i++)
{
sum += fft.getBand(i);
centroid += i * fft.getBand(i);
if ( fft.getBand(i) > fft.getBand(peak) )
peak = i;
}
centroid /= sum;
float peakiness = (float) peak / centroid;
///------------------------
noStroke();
fill(0,6);
rect ( 0,0, width, height );
translate(width/2, height/2);
fill(200,60);
for ( int i = 0 ; i < 10 ; i++)
{
float circlesize = random(70,100);
//ellipse ( random(random(-200,-20), random(50,200)), random(random(-200,-20), random(50,200)) , circlesize, circlesize );
ellipse ( random(-width/2, width/2), random(random(-200,-20), random(50,200)) * peakiness, circlesize, circlesize );
}
//filter(BLUR, peakiness);
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}Another day, another experiment. I tried to resurrect my Spectral Flatness code and I think it is a precision error in the geometric mean. I thought about scaling the FFT values but I will have to go back on the math for that to make sure I don’t skew the values the wrong way. For now I have used the arithmetic mean of the spectrum to give me some numbers to play with.
This sketch changes colour at different volumes, currently white for quiet and black for loud, and becomes more animated with noisier spectra (not really but close enough for now). I think I am getting close to something interesting. Maybe once I sleep on it tomorrow will show me the answer.
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
float lastlevel;
float[] myspectrum;
float[] features;
float spacing;
void setup()
{
size(720, 480, OPENGL);
hint( DISABLE_OPENGL_2X_SMOOTH );
smooth();
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
fft.logAverages(50, 2);
myspectrum = new float[ in.bufferSize() ];
noFill();
strokeWeight(8);
spacing = (float) width / fft.avgSize();
}
void draw()
{
background(30);
fft.forward(in.mix);
float sum = 0;
for(int i= 0; i < in.bufferSize(); i++)
{
myspectrum[i] = fft.getBand(i);
sum += myspectrum[i];
}
//==============================
// SPECTRAL FLATNESS
// DOESN'T WORK -- geometric mean always evaluates to zero :(
//==============================
float flatness = 0;
double geometricMean = (double) myspectrum[0];
float arithmeticMean = sum;
for ( int i = 1; i < myspectrum.length; i++)
{
geometricMean *= (double) myspectrum[i];
if (myspectrum[i] == 0.0)
{
println("WARNING: ZERO DETECTED " +frameCount);
break; // might as well leave since this thing will evaluate to zero
}
}
geometricMean = Math.pow(geometricMean , (1.0f / myspectrum.length) );
arithmeticMean /= myspectrum.length;
flatness = (float)geometricMean / arithmeticMean ;
//==============================
float level = in.mix.level();
// lowpass filter to smooth out motion
level += lastlevel;
level *= 0.5;
translate( width/2 , height/2 );
if (10 * log(level) > -50 )
stroke(0);
else
stroke ( 200, 40);
beginShape();
for ( int i = 0 ; i < fft.avgSize() ; i++ )
curveVertex( random(-spacing, spacing) * i, arithmeticMean * random(50,150) - 75 , random (0, -200) );
endShape();
lastlevel = level;
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}I guess I am at it again. Today I was playing around with an idea for making a spectral centroid visual. I was thinking about FFT displays and I thought that having bars rise and fall isn’t all that visually stimulating, though it is accurate for display. So, what I did was map spectral centroid to the location of a directional light in OPENGL and had it point to a line of boxes whose size modulates with the amplitude of the incoming signal. I also reversed the amplitude mapping so that loud sounds create small boxes. It makes for some nice motion.
I also spent some time looking through the Processing.org bug list in reference to some glitchyness in OPENGL as implemented in Processing. There are varying reports about how to smooth out anti-aliasing and threading issues. It seems the best thing to do is to try out all solutions and see which one works for that particular patch. Although the simplest thing to do to avoid the anti-aliasing problem is to only draw vertical or horizontal lines, but who wants to do that?
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
float r;
float zr;
float lastlevel;
float[] myspectrum;
float[] features;
void setup()
{
size(720, 480, OPENGL);
//http://code.google.com/p/processing/issues/detail?id=217&q=opengl&colspec=Stars%20ID%20Type%20Status%20Priority%20Owner%20Summary
//http://code.google.com/p/processing/issues/detail?id=53&q=opengl&colspec=Stars%20ID%20Type%20Status%20Priority%20Owner%20Summary
//noSmooth();
hint( DISABLE_OPENGL_2X_SMOOTH );
hint( ENABLE_OPENGL_4X_SMOOTH );
//smooth();
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
myspectrum = new float[ in.bufferSize() ];
fill(200, 40);
stroke(30,40);
}
void draw()
{
background(30);
lights();
fft.forward(in.mix);
for(int i= 0; i < fft.specSize(); i++)
myspectrum[i] = fft.getBand(i);
//==============================
// SPECTRAL CENTROID
//==============================
float m1 = 0;
float m0 = 0;
for (int i = 0; i < myspectrum.length; i++){
m1 += i * myspectrum[i];
m0 += myspectrum[i];
}
float centroid = 0.5;
if (m0 != 0.0)
{
centroid = (m1 / m0) / (float) myspectrum.length ;
}
//==============================
float level = in.mix.level();
// lowpass filter to smooth out motion
level += lastlevel;
level *= 0.5;
translate( width/2 , height/2 );
rotateX ( zr++ / 50 );
// set range of centroid to -1 -> 1
directionalLight( 200,100,230, centroid * 2 -1 ,0,-1 );
float spacing = (float)width / fft.specSize() ;
float boxsize = 10;
for ( int i = 0 ; i < fft.specSize() ; i++ )
{
pushMatrix();
translate( (i * spacing) - (width/2) + boxsize - 10 ,0,0);
box(boxsize - 10*log(level) - 30);
popMatrix();
}
lastlevel = level;
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}Today’s sketch is a play on the random shape clusters that I seem to be gravitating towards. I used the spectral feature extraction from Day 5 so not all of the code is posted below. You can go to that post and rip the features if you like.
This sketch uses a more coherent model instead of randomness. There are 20 3D boxes in the sketch that have their colour and placement changed based upon the sound coming into the sketch. It is reactive but a little too literal. I think I will try to make some mappings in the future that can take spectral centroids and do something meaningful visually. Sounds like a research project!
import javax.media.opengl.*;
import processing.opengl.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
Minim minim;
AudioInput in;
FFT fft;
float r;
float zr;
float lastlevel;
float[] myspectrum;
float[] features;
void setup()
{
size(720, 480, OPENGL);
hint( ENABLE_OPENGL_4X_SMOOTH );
minim = new Minim(this);
in = minim.getLineIn(Minim.STEREO, 512);
fft = new FFT(in.bufferSize(), in.sampleRate());
fft.logAverages(50, 2);
myspectrum = new float[ in.bufferSize() ];
//noFill();
//stroke(255,40);
fill(200, 40);
}
void draw()
{
background(30);
fft.forward(in.mix);
for(int i= 0; i < in.bufferSize(); i++)
myspectrum[i] = fft.getBand(i);
features = drawSpectral(myspectrum);
fill(features[0] * 200 + features[1] * 200 + 100, 40);
float level = in.mix.level();
// lowpass filter to smooth out motion
level += lastlevel;
level *= 0.5;
if ( 10 * log (level) > -50 )
zr++;
translate( width/2 , height/2 );
rotateZ ( zr / 50 );
for ( int i = 0 ; i < 20 ; i++ )
{
rotateY( r / 300 );
rotateX( r / 500 );
translate(log(level)*i,0,0); // comment this line out for more standard shape
box(width/4 + 100 + ( 10 * log(level)) +i*3);
}
r++;
lastlevel = level;
}
void stop()
{
in.close();
minim.stop();
super.stop();
}
void mouseClicked()
{
saveFrame();
}float[] drawSpectral(float[] spec)
{
//==============================
// SPECTRAL CENTROID
//==============================
float m1 = 0;
float m0 = 0;
for (int i = 0; i < spec.length; i++){
m1 += i * spec[i];
m0 += spec[i];
}
float centroid = 0.5;
if (m0 != 0.0)
{
centroid = (m1 / m0) / (float) spec.length ;
}
//==============================
// SPECTRAL ROLLOFF
//==============================
float perc = 0.8;
float[] sumWindow = new float [ spec.length ];
float total = 0;
float sum = 0;
float specRolloff = 0;
boolean specdone = false;
for( int i = 0; i< spec.length; i++)
{
sum += spec[i];
sumWindow[i] = sum;
}
total = sumWindow[spec.length-1];
for ( int i = spec.length-1; i > 1 ; i--){
if (sumWindow[i] < 0.8 * total)
{
specRolloff = (float) i;
specdone = true;
break;
}
}
if ( !specdone )
specRolloff = float(spec.length - 1);
specRolloff /= spec.length;
//==============================
// SPECTRAL COMPATCTNESS
// code snippet from jAUDIO
//==============================
float[] mag_spec = new float[ spec.length ];
for (int i = 0; i < spec.length; i++)
{
mag_spec[i] = abs( spec[i] );
}
double compactness = 0.0;
for (int i = 1; i < mag_spec.length - 1; i++) {
if ((mag_spec[i - 1] > 0.0) && (mag_spec[i] > 0.0) && (mag_spec[i + 1] > 0.0))
{
compactness += Math .abs(
20.0 * Math.log(mag_spec[i])
- 20.0 * (Math.log(mag_spec[i - 1])
+ Math.log(mag_spec[i])
+ Math .log(mag_spec[i + 1]))
/ 3.0);
}
}
float[] f = { centroid, specRolloff, (float)compactness };
return f;
}Got a little busy today so I made a quick sketch to see if I understood making shapes with OPENGL. I do!
import javax.media.opengl.*;
import processing.opengl.*;
int jf;
void setup(){
size(720,480, OPENGL);
hint(ENABLE_OPENGL_4X_SMOOTH);
fill(240,30);
}
void draw(){
background(0);
jf = int(random(10,50));
beginShape(TRIANGLE_FAN);
for (int i = 0; i < jf; i++)
vertex( random(0,width), random(0, height), random(0,100));
endShape();
}
void mouseClicked(){
saveFrame();
}
