3D audio, 3D audio spatialization, Ableton Live 9 Suite, digital audio workstation (DAW), Head-Related Transfer Functions (HRTFs), Oculus Audio Pack 1, Oculus Audio SDK, Oculus Rift, Oculus spatializer plugins, Oculus VST Spatializer, virtual reality audio, virtual reality headset, virtual world, VR
Peters was scrubbing his toes. I said.
‘Imagine hearing muffled footsteps behind you. You turn around and can now hear the footsteps perfectly clear. There is a man walking towards you with a hammer, ready to commit a bloody death.’
I continued. ‘Or imagine a jet plane flying over your head, ready to drop acid bombs upon your person.’
Peters squirmed. I spoke again. ‘Or a bee buzzing around your head, stinging you on the nose and cheeks a thousand times.’
My Dutch friend interrupted. ‘Actually, a bee will only sting once, then it will die.’
‘Okay,’ I replied, ‘It will sting you, and then you will hear it fall and splat on the ground.’
I was trying to explain the wonders of 3D audio on the Oculus Rift headset.
‘I already have top-notch audio in my house,’ Peters scoffed, ‘I don’t need no virtual reality headset.’
I told him that his two computer speakers could only ever give a flat sound. Right and left panning. Loudness.
But the Rift with headphones can provide 360 degrees of audio. I could be wearing the headset with a sound playing in front of me. I turn around in the virtual world and can now hear the sound coming from behind. Or above me, or below. To the left of me or to the right.
How can this be possible though? I mean, it seems simple enough to understand that a sound heard in our left ear before our right ear, that is louder in our left ear than in our right, must be a sound coming from the left direction. But how can we tell with a set of headphones that a sound is coming from behind us instead of in front of us? Or from above us, or below?
Turns out that HRTFs (Head-Related Transfer Functions) can help determine the direction of sounds, flying in from all angles. Indeed, we can also determine the distance of sounds with the help of reflections, reverberations and motion (as well as loudness, of course). The Oculus documentation provides an introduction to Virtual Reality Audio (including some excellent video talks by Brian Hook and Tom Smurdon).
All these techniques allow us to create 3D audio spatialization for the wearer of the Oculus Rift virtual reality headset – we can put a sound anywhere in their virtual world and it will stick into position. If the wearer suddenly turns away from the sound, they will hear it pass by their ear and continue to play from behind them. A wonderfully immersive experience of audio, as well as graphics!
The Oculus Audio SDK Guide covers the plugins available to programatically implement 3D audio spatialization for those lucky users of our applications. Using audio middleware such as FMOD and Wwise, or game engines such as Unity3D, we can track the user’s head orientation and position and plot our sounds.
But what if we want to try out 3D audio without the need to integrate it into a virtual reality experience or game? Well, it just so happens that there is an Oculus VST Spatializer for DAWs (digital audio workstations). So let’s give it a whirl…
Now for the amazing bit. I told Peters to put the virtual reality headset on, and take hold of the Xbox controller.
‘I am going to play some spooky sounds, which you will visualize as throbbing yellow balls in a 3D virtual world.’
I got the creepy horror sounds from the Oculus Audio Pack 1. It’s three full moons past Halloween after all!
Here’s the video of what Peters heard and saw, when wearing the Rift. Be sure to put on your own headphones when watching the video, if you want to experience the 3D audio!
The window on the left shows Peters exploring a virtual world of sound on the headset. The window on the right shows the spatializer settings being updated with Peters actions. Ableton is in the background, hosting the spatializer plugin and playing the creepy_blood_squish_slimy_03.wav audio.
We can hear the squishy blood sound in front of us. As we turn around, we can hear the sound in our left ear, behind us then in our right ear. We move above the sound and then below it. All directions!
We can move away from the sound and hear it in the distance. The Xbox controller can select the sound and move it about, changing its properties (Near / Far, and Gain / Distance if we so wish). The Y button on the Xbox controller provides us with a room for reflections, and the X button adds reverb.
Peters smiled. ‘You were right, it really is a new dawn for audio.’
And I knew he meant it, for he’d shat his pants with excitement. Didn’t even need the man with a hammer, or the bombs or honey bees. The sounds alone were enough.