As modern musicians, we have so much choice, maybe too much - we can create in so many ways, and there’s something for every budget, every mood, every environment. So why not throw VR into the mix as well? It’s never been more accessible, more versatile, or more deserving of a place in a music setup.
I’ve been exploring VR music-making for the last eighteen months, and I want to share what I’ve learned - this is my purely personal take. But before embarking on their journey into virtual reality, there’s something every user has to contend with. VR is one of the most cutting-edge ways to make music, but it’s a little controversial, partly because of its reliance on big corporations to provide the hardware, but mostly because of resistance to the frankly bizarre look of VR headsets: they take some getting used to. If you want to get involved, you’ll have to overcome these obstacles, but believe me, it’s worth it. It may be a little impractical, disorienting and even slightly dystopian - but making music in VR is like nothing I’ve tried before.
There is no history lesson today - suffice to say that if you want to read a comprehensive overview of the history of VR music, this is a good source. What I will point out is that there are different acronyms at play - VR (Virtual Reality, where you’d expect to be in a fully enclosing virtual space), AR (Augmented Reality, where you can still see the world outside courtesy of what’s called ‘passthrough’, with virtual objects superimposed on that), and XR (eXpanded Reality, which is more useful as a term because it encompasses all of the others). Let’s agree to stick to VR here, as it’s the most commonly used term.
Picture this: you’re wearing a VR headset, immersed in a landscape of your own design. You’re holding a game-style controller in each hand, running a synthesizer and sampler setup that’s floating in space around you. The headset’s built-in speakers are firing sound towards your ears. Good?
Now picture this one: you’re wearing a VR headset, but you’re able to see what’s going on around you, as built-in cameras let you view the outside. You’re using hand tracking to operate on-screen controls, but this time they’re connected to a computer running Ableton Live or Logic - the sounds emit from the computer, while the headset operates as a spatial control interface. With VR you can experience both of these scenarios - and anything in-between, with as much ‘reality’ as you want - using a simple rig made up of the Meta Quest 3 headset and the Patchworld VR app.
Before this started, I knew some people used VR for gaming, but that was all. I purchased a Meta Quest 3 out of sheer curiosity - the 3 was a good starting point, being the only version to offer full-colour passthrough. I searched for functional music-making apps, rather than rhythm games, and Patchworld grabbed my attention, with the familiarity of modular synthesis and sampling, combined with the use of visual elements. I’m a musician, but I’m also a photographer and (years ago) a VJ. Music and visuals are equally important to me, and when I learned that Patchworld could accommodate both, I couldn’t resist.
Patchworld is based on a collection of audio and visual tools, ranging from synths and samplers down to individual modular synth-style elements called blocks. Sampling is based on the visualisation of samples as floating bubbles, which works great in 3D space. Further instruments are available to purchase, and I’d recommend exploring the options here. The visual library includes skies, terrains, assorted 3D objects, and again breaks down into smaller elements where individual geometric shapes can be combined and grouped.
Patchworld can import audio samples, still images, GIFs, and even short videos. Every project you make can have a visual element. It’s possible to come up with something that looks minimal, but has a lot going on under the surface, because individual items can be tagged as ‘Hide In Play Mode’, so your jungle of blocks and virtual cables can simply disappear.
It was very easy for me to visualise what I wanted to do. Over the last few years I’ve been venturing further into beat-free music, creating moods and ambiences, and I’ve been a photographer, VJ, and filmmaker for a long time, so I was curious to see how all of this would fit together. This is what some people call ‘compositional performance’ - where you’re interacting with preconfigured elements rather than expressively playing a guitar or keyboard in real-time. I had an idea of the results I wanted, but no idea of how to get there.
Patchworld nudges users towards Discord as an answer for everything, but I prefer a well-written manual and solid technical support, so I continued blundering about. I’m still catching up with a few basics, but along the way I’ve been making music, and sharing it on YouTube, primarily through the PatchWorld YouTube channel. The point is, that although Patchworld can be fairly complex, you can begin with a few simple instruments, loops, and visuals. If you have the Quest 3, and Patchworld, you’re ready to create. The onboard speakers with spatial audio are good enough to get you going, and there’s a stereo headphone output. Everything else can be added later.
It’s best to jump in and start playing as soon as possible, you don’t need to wait until you’ve watched enough tutorials
The first project I made with Patchworld consisted of a 303-type sequencer, three blob-type samplers (two of them looping), a filter effect, and a virtual speaker. I also use the resizable mirror, which ‘reflects’ your avatar - it’s kind of fun. I used the Quest controllers, because at that point, hand tracking wasn’t available. As with every other project, I recorded the jam using the Quest’s native recording feature, which also captures audio as a stereo file, then moved it over to my Mac. The results were quite rough, but it got me started - with Patchworld, it’s best to jump in and start playing as soon as possible, you don’t need to wait until you’ve ‘studied’ it, or watched enough tutorials.
The most exciting part of all this is the way in which I can use real-world spaces in my projects. The most hardcore method, which I love to do, but is limited for obvious reasons, is to prepare a project at home, then go to a suitable location - an open space, an alley, a car park, whatever - and record the live performance there, with pass-through on. This results in a movie recording that captures the environment around me, the software elements I’m using, and the audio performance – and of course it’s live, so there isn’t much potential for editing afterwards. I love this, but as I mentioned, it’s limited, because there are only so many places you can go, put on the Quest, and get a recording done without anybody coming along to see what you’re doing.
The next best alternative, which is still pretty great, is to use 360º photos instead. These are a surprisingly effective way to ‘sample’ an environment. I’ve used an iPhone app that captures these images one step at a time, then stitches them together into a full 360º; and I’ve also used dedicated 360º cameras, which reduce the capture process to one simple click, although you’ll need to develop some tactics to avoid showing up in your own 360 images - they do capture everything, after all. By incorporating real-world spaces I can create moods and juxtapositions that I couldn’t otherwise. As well as capturing new material in new locations, I will go back into my archive and find stills, movie clips, drawings, anything that fits in with what I’m doing, building textures by collaging old and new material. I have Ableton Live projects going way back, even Minidisc field recordings - it’s all fair game.
Working with Patchworld on the Quest doesn’t mean you’re isolated, unless you want to be - it’s possible to incorporate it into a MIDI studio setup, and use it to run hardware or interface with a computer, connected wirelessly. Patchworld includes blocks that provide convenient routing for MIDI notes and CCs, while their PatchCompanion application handles things at the computer end. When I’ve needed specific sounds or effects, I’ve sent MIDI from Patchworld to either Ableton Live or Logic Pro; it’s very effective for ambient soundscapes, but the wireless connection can still encounter latency or timing drift, in my experience.
The isolation of a VR headset can be a benefit when creating, but limiting when you want to show off your finished piece - sooner or later you’ll want to get your creations into the outside world. It’s possible to post videos from the headset directly to social media, but I prefer to finesse them on a computer, editing the video with Final Cut Pro and audio with either Logic Pro or Ableton Live. One drawback with VR is that something can look and ‘feel’ amazing in the headset, but when it’s exported or captured as a 2D movie or still image capture, it can look flat and low-res - in an ideal world, everybody would have some way of experiencing exactly what you are, but that needs the audience to be wearing headsets as well. Aside from that, the Quest 3 supports USB audio interfaces, connected to the USB-C port on the left-hand side of the headset, and you can cast live video from the headset directly to the web or a projector.
I’ve found a workflow that suits me. I don’t do visuals, then sound, or vice versa - they happen together. I enjoy making these videos, and releasing the audio separately on Bandcamp is something that I’m exploring more now - using the recorded Patchworld jams as the basis of longer pieces - remixing them, I guess, is the closest to what I’m doing. Aside from releasing finished projects, I’d like to offer people a way to share and enter the VR world of my projects, as well as to stream live performances. Play a gig from the comfort of home, with no travelling and no setup? That would suit me.
Software that is dependent on specific hardware is always a risk. These are the chances we take when we embrace these things - it’s best not to be too loyal or too committed to any one platform. The world of VR is always changing, and subject to the corporate whims of those with enough resources to run it. I’m still using the Meta Quest 3, which was an absolute bargain as far as I’m concerned, although I hope to see other developers arrive on the scene for the benefit of healthy competition. At the moment it feels like Meta are more interested in AR glasses for the future, though time will tell.
The other big player is of course Apple, with its Vision Pro, but that is a rather expensive piece of equipment, and mostly focused on being a luxury home entertainment system - there’s nothing like Patchworld available for Vision Pro at present. Making music inside VR isn’t necessarily about abandoning your current setup, although it could be. It can be about complementing your setup - there’s no need to delete your DAW! In fact, Live or Logic or Reaper can function as a helpful add-on for a VR setup - you can send MIDI to it, of course, but also you need something to prepare samples and so on, and to handle recording and mixing. It’s not out of the question to commit to an entirely VR production and performance cycle, but that would be quite a leap of faith.
Do I recommend this type of setup as it stands today? Yes. You will look faintly ridiculous and quite possibly terrify your cat, but composing in VR can literally open up new dimensions of your creativity. And, not only is it a stimulating alternative to conventional production and performance, the truth is that most musicians that I’ve met have some visual flair as well (probably including you, if you’re reading this), and Patchworld is the best possible way to bring these two worlds together.
You must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.