Tags

, , , , , , , , , , , , ,

‘Life is like a hose,’ the wise old man said, ‘You can’t get water out unless you put water in.’

I nodded at his sage advice. But Arkwood replied, ‘Ah, a bit like sex then.’

I clipped Arkwood around the ear.

‘Never mind him,’ I said to the ancient beggar, ‘Please, tell me, how can I hear sounds drift upon the wind?’

Turns out all I need is a Microsoft Visual Studio C++ application with OpenGL graphics library and the Oculus SDK for Windows. And the Oculus Audio SDK. And the FMOD sound effects engine. Then the headphones on my Oculus Rift virtual reality headset can pipe 3D audio of an axe moving through the air into my lugs.

Let’s supply the FMOD API with C++ code to update the position of that axe. My post Oculus spatializer plugin for FMOD has some of the underlying detail (incl. exact API calls), so I will just concentrate on the general approach here:

aEngine.Init();

aEngine.LoadPlugin("OculusSpatializerFMOD.dll");
aEngine.LoadBank("Master Bank.bank", FMOD_STUDIO_LOAD_BANK_NORMAL);
aEngine.LoadBank("Master Bank.strings.bank", FMOD_STUDIO_LOAD_BANK_NORMAL);
aEngine.LoadEvent(eventAxe);

aEngine.SetRiftHeadphones();

After initializing the audio engine, I load the Oculus spatializer plugin along with the FMOD Studio output. The axe sound event is loaded (“event:/Axe”). And our Rift headphones are configured to receive the audio.

Here’s the axe sound event in FMOD Studio, and the plugin settings below it:

Lovely. But how do we play our axe sound event on each render cycle of our VR world?

Well, first we need to collect some info that the FMOD API will need to know about:

Matrix4f aFinalRollPitchYaw = rollPitchYaw * Matrix4f(EyeRenderPose[0].Orientation);
Vector3f aFinalUp = aFinalRollPitchYaw.Transform(Vector3f(0, 1, 0));
Vector3f aFinalForward = aFinalRollPitchYaw.Transform(Vector3f(0, 0, -1));

Vector3f eyePos0 = EyeRenderPose[0].Position;
Vector3f eyePos1 = EyeRenderPose[1].Position;
Vector3f eyePos = (eyePos0 + eyePos1) / 2;

Vector3f midEyePos = Pos + rollPitchYaw.Transform(eyePos);

We combine the yaw of our player and the Rift headset orientation (which is the same for either eye) and use it to yield vectors for Up and Forward. We also find our mid-eye position from both eyes, adding on our player’s position.

We also need to obtain our axe’s info in the 3D world:

bool axeActive = false;
Vector3f axePos = { 0.0f, 0.0f, 0.0f };
if (isIsland) {
	for (int i = 0; i < numMyAxes; ++i) {
		if (MyAxes[i]->IsActive) {
			axeActive = true;

			for (int i = 0; i < numMyModels; ++i) {
				if (MyModels[i]->MyModelName == Axe) {
					axePos = MyModels[i]->MyMeshes[0]->Pos;
					break;
				}
			}

			break;
		}
	}
}

Notice how I’m only interested in getting my axe position if it is active i.e. moving through the air. Our player also needs to be near the island which sports the axe, otherwise we are wasting processing time for a part of the landscape that we are not currently inhabiting.

Note all these values are in OpenGL world space.

Right, time to actually play the axe sound.

We give FMOD our player’s current Position, Up and Forward:

aEngine.SetListener3dAttributes(
	Vector3f{ midEyePos.x, midEyePos.y, -midEyePos.z },
	Vector3f{ 0, 0, 0 },
	Vector3f{ forward.x, forward.y, -forward.z },
	Vector3f{ up.x, up.y, -up.z }
);

The negation of the z axes values are important to get FMOD to work with the OpenGL right-handed coordinate system.

And then we can update our axe position, before playing the axe sound event:

if (axeActive) {
	aEngine.SetEvent3dAttributes(eventAxe,
		Vector3f{ axePos.x, axePos.y, -axePos.z },
		Vector3f{ 0, 0, 0 },
		Vector3f{ 0, 0, 0 },
		Vector3f{ 0, 0, 0 });

	if (!aEngine.IsEventPlaying(eventAxe)) {
		aEngine.PlayEvent(eventAxe);
	}
}
else {
	if (aEngine.IsEventPlaying(eventAxe)) {
		aEngine.StopEvent(eventAxe, true);
	}
}

Of course, if our axe is not active then we make sure the sound has stopped playing (which is what we also need to do if we leave our island area).

Always be sure to update the audio engine on each cycle, once our sound is set:

aEngine.Update();

So here is a video of the axe moving through the air – if you put on your headphones you will hear the chopping sound of the axe move from your right ear to your left ear, from in front of you to behind you, all in wonderous 3D audio.

Okay, so the chopping sound is shit for purpose. I snatched it lazily from the Oculus Audio Pack. The chopping sound isn’t even in time with the visuals, but hey it’s a proof of concept right?

The old man reached into his satchel and brought out a cupped hand of beans.

‘You have demonstrated how to place a person and a sound in perfect harmony. Have these beans, they will nourish you.’

Arkwood slapped the beans out of OAP’s wrinkled paw. ‘Fuck the beans, grandpa, where’s the Viagra?’

I held Arkwood down whilst the sage rung his ears with right and left hooks. Arkwood won’t be hearing no 3D audio for some time.

Ciao!

Advertisements