3D audio spatialization, cpp, FMOD API, FMOD Studio, Microsoft Visual Studio, Oculus Audio Pack, Oculus Audio SDK, Oculus Rift, Oculus Rift PC SDK, Oculus spatializer plugin FMOD, OpenGL, virtual reality headset, virtual world, VR
In my last post, my rotund Dutch lodger Peters had a shot of the Oculus VST spatializer for DAWs (digital audio workstations). He put on the Oculus Rift virtual reality headset and wondered at the 3D audio.
‘I can walk around the squishy blood sound, hearing it in front of me, to the left of me, behind me and to the right. Man, I can even hover above the sound or duck below, stroll away from it or wrap it in a room full of reflections and reverberations!’
But what if we could drop 3D sounds into a virtual world of our choosing? The Oculus spatializer plugin for FMOD lets us do just that, to programmatically implement 3D audio spatialization for those lucky users of our applications.
The C++ application that I am going tinker with is the RoomTiny OpenGL Visual Studio project available at the Oculus Developer Downloads page, part of the Oculus SDK for Windows. The downloads page also provides the Oculus spatializer plugin for FMOD, part of the Oculus Audio SDK Plugins. And if that wasn’t enough, we can also get our mitts on some creepy sounds courtesy of the Oculus Audio Pack!
Okay, first up, let’s visit the FMOD download page and grab the Windows 64-bit FMOD Studio Authoring Tool. We’ll also be needing the Windows FMOD Studio Programmer API and Low Level Programmer API. I’m using Version 1.08.14 of both.
Here’s FMOD Studio housing my Squish and Creak sound events, set to loop and added to the Master Bank.
Notice that I’m using the Oculus Spatializer and Oculus Spatial Reverb plugin effects. The Oculus Audio SDK documentation provides an Oculus Spatializer for FMOD Integration Guide to help us add the 64-bit Oculus spatializer plugin for FMOD into FMOD Studio and configure it.
Once the Master Bank has been built, we can drop its ‘Master Bank’ and ‘Master Bank.strings’ output into our application folder. We’ll also drop a copy of the Oculus spatializer plugin for FMOD into our application folder (32-bit or 64-bit dll, as appropriate).
So now we can crank open the RoomTiny OpenGL Visual Studio project and integrate our FMOD middleware framework. I’m using the free Microsoft Visual Studio Community 2015.
To get the hang of how to call the FMOD Studio API (and the Low Level API) I’d suggest reading the excellent Making a Basic FMOD Audio Engine in C++ article from Cody Claborn.
Cody’s article contains a link to his FmodStudioEngine on GitHub. The FmodStudioEngine wraps the FMOD API and provides initialize and update functions, along with a host of functions to handle the loading and playing of sound files and events.
Cody even has a Setting Up Xcode and Visual Studio for FMOD Development article, to help us integrate FMOD into our projects.
After we have initialized the FMOD studio system, and before we load our bank files, we will need to load our Oculus spatializer plugin for FMOD (via the low-level system):
We can then load our Squish and Creak events, and call their set3DAttributes function:
The f3dAttributes parameter is of type FMOD_3D_ATTRIBUTES, allowing us to specify where the sound should be placed in our 3D world.
We can then play the event and update the studio system.
But how do we tell FMOD where we are standing in the virtual world, so that it can play sounds in front of us, to the left of us, behind us and to the right? We need to call the setListenerAttributes function (via the studio system):
Again, we pass a parameter of FMOD_3D_ATTRIBUTES, so we can tell FMOD our Position, Forward and Up in the virtual world.
We can extract the Position, Forward and Up on each frame render via an invert of our View matrix. Michaelangel007 on Stack Overflow shows that the Forward X Y Z coordinates can be obtained from the first three values of the matrix’s third row, and that the Up X Y Z coordinates can be obtained from the first three values of the matrix’s second row. The Position X Y Z coordinates can be obtained from the first three values of the matrix’s fourth column (12, 13 and 14 on his diagram).
The RoomTiny project coordinates require me to re-invert the Position Z to get things right:
const float position_z = -(view.M);
Remember to update the studio system on every frame render too.
Great. Now, when we pop on our Oculus Rift headset and stroll about in our virtual world, FMOD – and thus the Oculus Spatializer – will know exactly where we are. The Squish and Creak sounds will stick to their specific place in the 3D landscape. We can hear them as we pass by – with whatever reflection and reverberation we grant them – dancing from one ear to the next.
Here’s a short video of me walking around the sounds. Be sure to pop your headphones on, to hear the 3D audio spatialization!
Not the most pleasant of sounds, but you get the idea.
The Creak sound has been placed near the top of the blue chair and we can hear it in front of us, to the right and left and behind us, as we move our head about and stroll around the table.
Over in the corner of the room is the Splash sound, which seems to be emanating from thin air. There is no object representing the sound, yet we can still place it with our ears alone. That’s the power of 3D audio spatialization!
At the end of the video we wander away from the creepy sounds, hearing them fade into the distance.