Jack and the StyloSim

My next programming project was designed as an experiment to see if Pure Data could deal with QWERTY keyboard input, USB input and audio input all at the same time.

I planned to make an audio effects device, using a Stylophone for the audio input, plugged into the laptop’s 3.5mm (1/8″) stereo mic/line in socket; the laptop keyboard; and a nice twin-joystick device I found on eBay.

The joystick device was called ‘USB Simulator’ (‘For Airplan Heli’) and had the model number FS-SM020.  The two joysticks were quite good to use – not the kind you grip in the hand, but bigger than the thumb-operated joysticks you get on a game controller, plus each of the 4 axes had separate adjusters which you could use to fix the centre of the joystick’s range of control.  There were a couple of buttons on the front, which you can see on the bottom right and left, but these proved to be cosmetic only once I opened the case and looked inside.

I couldn’t find out much about this device on the internet, except that it was, as its name implied, normally used for flight simulation – presumably to simulate a radio-control transmitter – and, in fact, it came with a CD, probably containing the simulation program.

The case was quite large, and there would be ample room inside for further circuitry.  I may look at this possibility later, but for the moment I made no alterations to the device, intending just to use the two joysticks: a bandpass (‘wah wah’-type) filter and volume for the left one, and a binaural panning control for the right one.

The first problem to be overcome was to make sure the audio input from the Stylophone couldn’t be heard, only the output from Pure Data, which would hopefully be the audio input modulated by the joystick effects.  This was achieved by the simple-to-use and very handy audio-routing program Jack.

Jack works by running an app called JackPilot.  When opened this presents a small window like this:

Actually, the top button on the left reads ‘Start’ when opened, but I took this screenshot after clicking it and starting the Jack server.  The normal procedure is to open Jack, start all the programs involved in your routing scheme, then click ‘Start’, and then, when it’s ready, ‘Routing’.  (Just occasionally, you may have to start some audio running through a program before Jack recognises it, but mostly that isn’t necessary).

The Routing window looks like this:

All the inputs are in the left-hand column, all the outputs on the right; it always shows the System audio in and out.  In this instance I was just practising, so I only used Pure Data, but I would frequently have Logic running, too.  To set the route, click on all the inputs and outputs in turn (they are shown in red when being edited), and click on the item(s) you want them to be connected to before moving on to the next one.

At the beginning of the Pure Data program I put the ‘Print’ and ‘Open’ instructions, as I did for the Theresynth and Cybersynth (see previous posts):

This configuration enabled me, once I’d plugged in the Stylophone (audio input) and Simulator (USB input),to check the Device number of the Simulator and open the Device so the ‘hid’ object would receive and deal with the output from the two joysticks.  When I clicked the ‘Print’ message box at the top, the following was printed in the Pure Data window, telling me that the Device Number was ‘0’:

(It also gave the Device’s name ‘FS-SM020’, which it hadn’t done with the particular game controller I used for the Theresynth and Cybersynth).

As the Device number was 0, I amended the ‘Open’ box to show this number, and clicked it.  Then I clicked the ‘Print’ message box at the top again, and it showed me the controls it could recognise:

The 4 axes of the joysticks can be seen (‘abs_x’, ‘abs_y’, ‘abs_z’ and abs_rx’, I think, and I can’t remember if any input was recognised from ‘abs_ry) – and also 3 buttons, but the button function hadn’t been implemented in this device.

I clicked the on/off toggle switch, applied the stylus to the Stylophone keyboard, and sounds issued forth – a very good sign!

The Pure Data StyloSim program had three inputs:

On the left is the [key] object, which receives input from the laptop keyboard.  In the end, I added only the box with [== 114] in it: 114 is the ASCII code for the ‘r’ key, so the program only responds to the letter r, turning a reverb effect on or off.

[Edit: I later changed this to 119, the letter ‘W’, to avoid conflict with another program that used ‘R’ for a different function]

In the middle is the audio input, [adc~] (analogue-to-digital converter), which leads to the output stage.

On the right is the [hid] (Human Interface Device) object, receiving input from the 4 axes of the joysticks.  The [route] objects separate the 4 inputs and send them in different directions for their various functions (bandpass filter, volume and binaural panning).

All the effects worked – although the panning was less binaural than left-to-right; I’ve heard better.  This is the complete patch, showing the objects I used, and the mathematical calculations required to get it right.

The patch can be downloaded from here.  (Click ‘Save Link As’).

This proved that Pure Data could combine information from the keyboard (or an external USB QWERTY keyboard) and [hid] to work on an audio signal entering simultaneously.   As  can be seen from the following picture of the set-up in action, I used ‘The Alien‘, my first Stylophone modification, as the audio source.

This instrument has a ‘drone’ function, so it’s possible to put the stylus down and operate both joysticks on the Simulator at the same time.  Both the joysticks, incidentally, are centre-sprung, meaning that all values return to the central ones, except the y-axis of the left one – the bandpass filter.  This means the centre-frequency of the filter can be set and left while other parameters are adjusted.

Here’s a sound file illustrating the StyloSim:

It’s more of an example than an actual piece, just to show what the StyloSim does.

It’s slightly edited, but only to make it a bit shorter, otherwise it’s exactly as it sounded when played. There’s no post production effects on it, I just hit the ‘r’ button to turn on the StyloSim’s reverb before I started.

There are three parts: the first part is the Stylophone played with the stylus in the conventional manner; the second part uses The Alien’s ‘drone’ feature and variable pitch and vibrato controls; the third part is played with the ‘feedback’ switch on.

[Edit: more modifications have been made to the StyloSim joystick (although not for this particular use).  See this post].