Audio is a fundamental part of VR Immersion.
Brian Hook, Oculus Audio Engineer
Video: An introduction to Audio in Virtual Reality – with Oculus Rift’s Brian Hook
Audio in VR is a very important pillar in helping seal the experience and help create the feeling of ‘Presence’ in the virtual world. Sound enhances the perception of size, space and distance and glues the visual scene together. In Hook’s 2014 talk at Oculus Connect, he spoke in particular about two things:
This is the synthetic inverse of localisation which refers to a listener’s ability to identify the location or origin of a detected sound in direction and distance, it is where sound is designed to mimic that of reality but is actually artificial. In human sound localisation, we distinguish sound based on the lateral, elevation, front/back, pinna, head/shoulder/torso and distance values to evaluate where they are coming from and what they are.
Basic methods of spacialisation typically used in speaker systems for games and films adapts crude approximations constructing sound based on the left/right pan, front/back lower pass filler, distance volume attenuation and zero head tracking information. However, VR uses advanced techniques to induce spacialisation using direction HRTFs, distance i.e. direct/reflection ratio and volume, as well as the full utilisation of head tracking.
When it comes to handling ambience Hook notes that VR is in need of improvment, however, it does provide the participant with indirect audio cues about location. Ambience needs indirect sound propagation like early reflections, late reverberations and absorptions from the environment. At the moment it is crudely simulated, but to fully replicate this in VR more research needs to be explored into the geometry and material reactiveness to sound in a virtual space.
Sound Design: Tips on Mixing Scenes.
- Avoid listener fatigue i.e. too much going on all the time.
- Slower sounds have fewer artifacts.
- Audio should match visuals.
- Static ambient sounds that aren’t positional can be disconcerting.
- Not everything needs spatialisation.
- In-head narration/UI
- Low frequency rumbles
Spatialisation is best experienced with closed back Headphones for isolation in the VR environment.
Performance Implications in using spatialisation methods can result in the following:
- Takes up CPU
- Effects latency on graphics and audio, between 3-100ms)
- Use audio format that you are placing right from beginning
- Prioritise sound for spatialisation
- choose N most important
- Avoid spatialising LF sources
- Allow for ‘2D positioning’ path
Real Space 3D Audio
As part of Oculus’s Audio Initiative, VisiSonics is behind the development of Real Space 3-D Audio. They are self-described as leveraging world-class scientific discovery to add a third dimension to the sound you hear over headsets, bringing 3D sound to gaming, virtual reality, and entertainment (movies and music) platforms.
RealSpace™ 3D Audio is the virtual placement of sound anywhere in 3-D space with pin-point accuracy, creating the perception of real source direction, distance, depth, and movement relative to the listener and heard through standard stereo headphones. It recreates the auditory ambience of the environment, creating a complete immersive audio experience. VisiSonics’ developed the proprietary science-based algorithms and hardware giving the listener the experience of actually being present in the scene.
To create a convincing artificial audio environment for a user wearing headphones, sound processing must faithfully reproduce the acoustic spatial cues which would have been received at the ears in a real-world environment. RealSpace 3D achieves high resolution by including proprietary adjustments necessary to accommodate the complex interplay between the two cues.
The brain processes audio and visual information with equal weight. By combining 3D sound with virtual reality technologies, a user is provided with an immersive experience that reflects the world more accurately than ever before.
Below is a demo build from that fully illustrates the effectiveness of this process for virtual reality.
Real Space 3D Audio for Unity can be downloaded here: http://realspace3daudio.com/unityplugindownloads/
The Unity plug-in seems like it being well maintained as updates and developments are being documented through their Blog which can be seen at: http://realspace3daudio.com/blog/ along with Unity Forum Support which can also been seen at: http://realspace3daudio.com/blog/
All in all, this audio package for unity has had pretty good reviews and is definitely something to test while developing the sound design of our project.
There is also Oculus Audio SDK, which there is a little less documentation on.
Sources: Accessed on the 18th of February, 2016.