The Theresynth

I recently blogged about using a gamepad as a drum controller (https://andymurkin.wordpress.com/2012/01/09/drums-with-multicontrol-v0-4/), and this was my next project using a Human Interface Device (HID).

HIDs are the devices we use all the time to interact with computers.  Keyboards, mice, trackpads, trackballs, joysticks, gamepads/joypads/game controllers and graphics tablets are all HIDs, and all can be turned to musical use with the aid of a suitable program to interpret their output.  I chose to start with projects using game controllers because they have a fair number of buttons – and often joysticks – and are designed to fit comfortably in the hand (or hands).

The gamepad I used for the drum controller had just buttons, but the one I used for the Theresynth had two thumb-operated joysticks.  I planned to use one of these for controlling the pitch of an instrument.  In addition, although I had used the pre-made MAX/MSP app ‘MultiControl’ for the drum controller, I planned to use Pure Data and do my own programming.

This was not as reckless at it seems, as programming in Pure Data is done graphically.  Many mathematical and audio functions are pre-programmed for you: all you have to do is draw boxes, write in them what function you want performed, and join the boxes together in sequences.  Easier said than done, as I discovered, but easier, I thought, than writing lines of code.

I have described progamming in Pure Data in my post about creating alternative musical keyboards, using QWERTY keyboards (https://andymurkin.wordpress.com/2012/01/01/alternative-keyboards-4/).  There, ‘key’ was the main Pure Data function used, as this is designed to receive key presses from the QWERTY keyboard; in the application I’m about to describe the special function ‘hid’ was the key to interpreting the output from the game controller.

There are many Pure Data instructional videos on YouTube, and this one in particular: www.youtube.com/watch?v=HB_oVny33wA deals with ‘hid’ and game controllers.

If you’re working on a project like this one, the first thing to do is to allow ‘hid’ to get Pure Data to recognise the game controller.  To do this, plug in the game controller, run Pd-Extended.app, open ‘New’ from the ‘File’ menu and write (or draw, rather) the following into the empty box ‘Untitled-1’.

This is a short program which will show all the connected USB devices which Pure Data can recognise.  The ‘Print’ command is in a ‘message’ box, and is connected to the input of the ‘hid’ object.  Inputs are on the top of object boxes; outputs are on the bottom.  When you click ‘Print’, it sends the message, and gives a reply like this in the Pure Data window:

Each connected device has been given a number.  The first six, ‘Device 0’ to ‘Device 5’ are simply the different elements of my MacBook, keyboard, trackpad, and so on; but the seventh one, Device 6, looks like the important one: ‘4-Axis, 12-Button with POV’, ‘device type: joystick’.

Sometimes it will tell you the make and model; in my case, my ‘PC Line Rumble Game Pad’ is identified only by its ‘vendor ID’, 0x0583, and its’ product ID’: 0xa009.

We need to know the Device number, as we’ll use that in the program.  It doesn’t always come up as Device 6, so I always include in my Pure data programs a facility to check this number, and amend it if required.

The next step in writing a Pure Data program based on the ‘hid’ object, is to identify the outputs from the buttons and joysticks  in the controller.  This is done by a couple more additions to the simple program which produced the list of devices:

‘Open 6’ is another message box.  The small square on the left-hand side is an on/off ‘toggle’ switch, which I won’t be using now, but I’ll use it in a minute.

Click ‘Open 6’, and messages from Device 6 are routed to ‘hid’; click the ‘Print’ message box at the top again and the ‘Print’ object at the bottom will produce a list in the Pure Data window of all the buttons and controls in the device:

This shows 12 buttons, (‘btn_0’ to ‘btn_11’), the up-and-down and left-to-right axes of the left-hand joystick (‘abs_x’ and ‘abs_y’), the up-and-down and left-to-right axes of the right-hand joystick (‘abs_rz’ and ‘abs_ry’) and a hatswitch.

Now you know this, you can separate each of the inputs as they come out of ‘hid’ and get them to do various things – make noises (as I did with the gamepad MIDI drum controller), control effects (like filters, for example), and so on.  Other Pure Data objects (boxes like ‘hid’ and ‘print’) can do mathematical transformations along the way.

The way to separate them is in stages.  First of all, look at the left-hand column above.  There are two types of entry here: ‘abs’ and ‘key’.  Using a Pure Data object called ‘route’, we can get the ‘key’ information (from the buttons) to be separated from the ‘abs’ information (from the joysticks and the hatswitch):

Then, looking at the second column, we  can separate out the 4 joystick axes and the 12 buttons:

While I was testing this out just now I did two extra things, which you can see in the screenshot above: firstly I added the small box at the bottom, which is called a ‘bang’.  If everything’s working properly, this will flash on and off when I press the button ‘btn_0’ – which is very likely marked ‘Button 1’ on the game controller – and secondly I clicked the on/off switch (the cross means ‘on’) so that ‘hid’ would recognise and output button presses and joystick movements.

Because the MIDI note number and control system lends itself easily to mathematical manipulation, this  would be a common way to use Pure Data.  In this particular application, however, I decided to use some of its many audio functions.

I’d spent some time modifying stylophones, as you may have read in previous posts.  The thing about stylophones is getting great sounds from them is easy, but getting sinewaves out of them is well-nigh impossible.  In Pure Data, however, it is extremely easy, so I decided to create an instrument which would produce ethereal sinewave sounds, rather like the Theremin.

The Theremin is a wonderful electronic musical instrument, which you play without touching it!  If you don’t know what it is, you can read about it on the Wikipedia at http://en.wikipedia.org/wiki/Theremin.

My plan for the ‘Theresynth’ was less ambitious, but would involve using one of the game controller joysticks (the left one) to control pitch and the other (the right one) to control volume.  In the event, I also added a kind of vibrato (more of a fast auto-pan), controlled by the other axis of the right-hand joystick.  This added a little more depth and interest to the sound.  As well as this, a programming error I made along the way also enabled me to add a second tone to the instrument, which was a kind of frequency modulation/ring modulation – very effective in the lower register.

The exact method by which I achieved this is rather lengthy to explain, and I’ll attach the ‘patch’, as bits of Pure Data programming are called, so you can see, if you want to study it.  In principle, though, it went like this:

The output from the the joysticks is in the form of numbers from 0 to 255.  Divide this in half and you get all the MIDI note numbers (they run from 0 – 127); convert the MIDI numbers to frequencies and use this as the input to an oscillator, feed the oscillator into an output amplifier and you have a simple musical instrument!  All these functions are available in Pure Data.

Using ‘abs_x’ (left-to-right on the left-hand joystick), as I did in the Theresynth, that part of the circuit would look like this:

‘/ 2’ means ‘divide-by-2’; ‘mtof’ means ‘convert MIDI note to frequency’; ‘osc~’ is an oscillator; and ‘dac’, the digital-to-analogue converter sends audio out to the computer’s sound system.  The number 64 is is in a ‘number’ box – it doesn’t do anything, it just tells you what MIDI note is passing from ‘/ 2’ to ‘mtof’.  Useful if you want to check what’s going on.

This program works – I’ve just tested it (but don’t forget to turn ‘audio ON’ from the ‘Media’ menu of Pure Data) – but the problem is that the travel on a gamepad joystick is not that far, and you don’t want to go from the lowest MIDI note possible to the highest in such a short space.  So a lot of what I had to do was to restrict the range of notes available.

In the end, as you’ll be able to see in the finished patch, I did this by dividing again by 10, severely restricting the range of notes available to the joystick at any one time; and then using one of the buttons as a freeze-pitch button, with a very short time-delay to allow the joystick to spring back to the middle.  In this way it’s possible to play a few notes, ending higher or lower than where you started, freeze the pitch there, let the joystick go back to the middle at this new pitch, and gradually move up and down in steps, as required.  After a little practice, this came to seem quite natural.

I never quite got the hang of creating a nice interface for my Pure Data programs, so the window that opens looks like this:

Following these steps gets the Theresynth ready to use.  Steps 1 – 4 just operate the checks I described above: identifying the device number, checking the buttons are all recognised by the program, opening the device and switching on ‘hid’.  Step 5 reminds me to turn the audio on – although in later programs I found out how to do this automatically when the program loads.  Step 6 may or may not be necessary, but clicks some toggle switches on and off to guarantee everything is ready to work, but stays silent.  Step 7 turns up the volume and you hear the initial tone, which is A (440Hz).

The actual programming is visible in two windows, ‘controls’ and ‘oscillators’, which can be opened by clicking the boxes in the bottom right.

In my next post, I’ll describe how I improved this design, but the instrument definitely worked, and seemed to me to be a good use for a second-hand joystick.