socketMusic tech breakdown
back to music/art

What is socketMusic?


socketMusic early demo


socketMusic is a way to deploy and trigger audio on internet-enabled devices without the use of wires or any sort of special technology (except for a router). Think of it as a multichannel speaker array, except other people's cell phones are your speakers. Its inspiration comes from laptop orchestra

It uses node.js to host a webpage, its modules express and socket.io to target individual devices, the node-osc module to trigger functions in my node app via osc, and DNS server software such as dnsmasq to allow clients to connect to my node app without having to type in my ip (note that this requires configuring the hosting computer to use a static ip). Finally, I use audio programming synthesis environments such as Pure Data or SuperCollider to send OSC triggers to my node app and also generate sound to be played over larger main speakers. You can find an overview of the setup here.

File Manifest

My node application for my piece socketMusic:wireless contains the following files:
RUN/HOSTED BY NODE:
index.js - the file run by node hosting the html page, managing socket connections, and bridging the gap between the triggerer (myself/SuperCollider) and the triggerees (the clients accessing the html page)
index.html- the html page hosted by node
javascript/client.js - the script file included by index.html that contains the responders to the socket broadcasts from node (socket.on functions)
javascript/audio.js - the script file included by index.html that contains all the web audio api stuff that actually do the sound-making on the devices

RUN ON SUPERCOLLIDER:
frontend.scd - the SuperCollider file that sends OSC messages to my node app to trigger socket broadcasts, triggering the socket.on functions client side

index.js

The heart of socketMusic, what makes the whole concept work, is the node application running on my computer. The top of index.js declares function names, lists port numbers imports the necessarily modules (express, http, socket.io, node-osc).

//declare functions
var broadcastExample;
//declare ports
var socketPort = 80;
var oscPortIn = 33333;
var oscPortOut = 11111;
//import modules
var express = require('express');
var app = express();
var server = require('http').Server(app);
var io = require('socket.io')(server);
var osc = require('node-osc');


Note that my node application is listening on port 80, which is typically reserved for http and cannot be used by node. On Linux, I used the setcap command to allow node to do so.
setcap 'cap_net_bind_service+3p' /usr/bin/nodejs
The next bit of code initializes OSC client and server, using the oscPortOutport on localhost for the client and a null IP with the oscPortIn port for the server. I then setup the OSC server to deal with incoming messages and then trigger functions. I respond to "/clients" by listening the number of clients connected and shipping off the list of all connected clients (I'll show how I do that later on) back to SuperCollider (which sent the "/clients" message in the first place) on the same address.

var oscClient = new osc.Client('127.0.0.1', oscPortOut);
var oscServer = new osc.Server(oscPortIn, '0.0.0.0');

oscServer.on("message", function (msg, rinfo){

if(msg[0] == '/clients'){
	console.log(clientCount);
	oscClient.send('/clients', clientList);
	};

if(msg[0] == '/broadcastExample'){
	broadcastExample(msg[1], msg[2]);

};


};


In the next bit of code, I deal with routing requests that the clients make to my node/express app. "__dirname" poins to the directory index.js is running in.
//basic routing
app.get('/', function (req, res){
	res.sendFile(__dirname + "/index.html");
});


app.get('/javascript/client.js', function(req, res){
	res.sendFile(__dirname + '/javascript/client.js');

});

app.get('/javascript/jquery-2.1.3.min.js', function(req, res){
	res.sendFile(__dirname + '/javascript/jquery-2.1.3.min.js');

});

app.get('/javascript/audio.js',function(req, res){
	res.sendFile(__dirname + '/javascript/audio.js');

});

app.get('/css/style.css', function(req, res){
	res.sendFile(__dirname + '/css/style.css');

});

app.get('/media/sample.wav', function(req, res){
	res.sendFile(__dirname + '/media/sample.wav');

});



Next, I start up the node/express application.
server.listen(socketPort, function(){
	console.log('listening on ' + socketPort);

});


In the next bit of code, I deal with managing the sockets and clients. The dictionary "clients" stores sockets by their socket id. The array clientList storse a list of client ids and clientCount stores the number of clients. Each of these three variables are modified upon a socket connection and disconnect.
var clients = {};
var clientList = [];
var clientCount = 0;

//called on connection
io.sockets.on('connection', function(socket){
	clients[socket.id] = socket;
	clientList.push(socket.id);
	clientCount++;

	socket.on('disconnect', function(){
	clientCount--;
	delete clients[socket.id];
	var clientIdx = clientList.indexOf(socket.id);
	if(clientIdx > -1){
		clientList.splice(clientIdx, 1);
		
	};

	});

});http://socket.io/get-started/chat/



Finally, I define the functions called by the incoming messages which in turn broadcast to all sockets (io.sockets.emit) or target a specific socket (io.to(target).emit).

//target is passed through msg[1] in the incoming OSC message and arg through msg[2]
broadcastExample = function(target){

//broadcast to all targets
//the parameters following the event name clientFunction  are arguments
if(target == "all"){
	io.sockets.emit("clientFunction", msg[2])
	};

//target a specific socket
else{
	io.to(target).emit("clientFunction", msg[2]);
};

};


index.html

index.html is what the client will be loading when connection to my node app by typing in "socket.music" (which is directed to my static IP through my locally hosted DNS server) and serves as the front end of what the client sees. As the web audio api stuff is called from client.js, it's important to include it first. On my page, I have simple directions and a button which calls init() in audio.js which intializes Web Audio API and loads all the sounds. On an iPhone, there MUST be a user-action before the phone can make sound and data is loaded. It's annoying, but understandable as to prevent huge data rate spikes (which doesn't apply to my piece anyways since it's all over WiFi). I won't go over this here, but I use jQuery to change the text in the button to indicate things are loaded and I also use it to change the text at the bottom of the page indicating the section the piece is in.

client.js

Getting back to socket business, client.js contains responders to the socket events broadcast by index.js and lives on the client side.

//establishing sockets on the client side
var socket = io();


//argt comes from msg[2] on the server side
socket.on('clientFunction, function(argt){
	makeNoise(argt);

});

audio.js

audio.js on the client side contains all the code for the web audio api stuff. I won't go over it in too much detail here because there are plenty of web audio api tutorials online. I start off by declaring my globals such as the audio context and various bits that should be global, such as buffers and gain nodes and oscillator nodes. Then, I declare my aforementioned init() function that sets the web audio api context up and calls other functions that set up oscillators and sound file buffers.


var context, sinGain, sinOsc, soundBuf, soundOsc, soundGain, soundDur;

function init(){
	if(!context){
	try {
		window.AudioContext = 
			window.AudioContext || window.webkitAudioContext;
		context = new AudioContext();
		}
	catch(e) {
		alert('Web Audio API is not supported in this browser');
	}
	sinInit();
	soundInit();
	}
};


var sinInit = function(){

	//sinOsc init
	sinOsc = context.createOscillator();
	sinGain = context.createGain();
	sinOsc.type = "sine";
	sinOsc.frequency.value = Math.random()*5000+500;
	sinGain.gain.value = 0;
	sinOsc.start(0);
	sinOsc.connect(sinGain);
	sinGain.connect(context.destination);


};


var soundInit = function(){
	var req = new XMLHttpRequest();
	req.open('GET', 'media/sample.wav', true);
	req.responseType = 'arraybuffer';

	//decode async
	req.onload = function(){
		context.decodeAudioData(req.response, function(buffer) {
			soundBuf = buffer;
			soundDur = buffer.duration;
			}, function(e){"Error with decoding audio data" + e.err});

		}
	req.send();
};

//called by event 'clientFunction'
var makeNoise = function(argt){
	var now = context.currentTime;
	sinGain.canceldScheduledValues(now);
	sinGain.setValueAtTime(1.0, now);
	sinOsc.frequency.setValueAtTime(argt, now);
	sinGain.setValueAtTime(0.0, now + 1.0);

};

frontend.scd

On the server side, I use SuperCollider to send OSC messages and get clients lists and also generate sounds. I'll only include bits of the helpful stuff pertinent to learning how the socketMusic concept works. I store the port number node is listening to for OSC in the variable b and I create an OSCdef to receive messages from node about clients, storing the result in the ~clients environmental variable.

b = NetAddr("127.0.0.1", 33333);


//use this to receive messages from node with the '/clients' address
OSCdef(\receiveClients, { arg msg, time, addr, recvPort;
//recvPort is the port node is sending OSC messages out on
		~clients = msg[1..]; ~clients.postln;}, 'clients', recvPort: 11111);

//use this to poll node for clients
b.sendMsg("/clients");

Concluding Remarks

That should about cover the framework of what makes my socketMusic concept work. My first piece with the concept, "socketMusic: wireless", premiered at Stony Brook University on May 4, 2015 as part of my final doctoral recital. I posted a link to the soundcloud entry in a previous post.
link to boiler plate on github