The Phantom in the Opera Theatre

Story: Robert Clark

WAVE-FIELD SYNTHESIS EXPLAINED

As a mathematical formula, Wave-field Synthesis has been around for a few hundred years, but it has been evolving into a complex algorithm for increasingly practical use since the late-eighties. Basically it’s all about recreating wave-fields to suggest imaginary sound sources, which allows you to replicate spatial circumstances that don’t actually exist. One company that specialises in the advancement and application of wave-field synthesis is IOSONO, a small R&D firm established by the Fraunhofer Institute for Digital Media Technology around 10 years ago. These days it concentrates on streamlining the usability and efficacy of the theory in everyday audio situations via customised software and hardware.

Stephan Mauer is the Manager of Application and Product Planning at the German-based company, and he has an elegant analogy to explain what wave-field synthesis achieves. “Just imagine a smooth water surface and you throw in a stone, then you have the circular waves around it, and imagine at the shore of the beach you have a certain pattern of the waves as they approach the shore. And just imagine that at the shore there are loudspeakers and everything you see on the water is purely virtual. And what we do with the loudspeakers is just compute what every loudspeaker should do to maintain this wavefront. So, there is no sound behind the speaker but we think of a virtual sound source and ask what the speakers would have to do in their position in order to recreate the sound field that would have been created by this virtual source. In other words, I don’t have a violin there, but I want to perceive it in that position, so I need to re-create the wave front that the violin itself would create; that’s what the algorithm knows how to do.”

As it turns out, Stephan Mauer and Ralf Zuleeg had already been collaborating on projects using IOSONO’s technology in live venues around Europe. Zuleeg: “This collaboration between IOSONO and d&b started at the end of last year, when we began to discuss whether the wave-field synthesis algorithm could really work in the live-sound world. I put one machine in a club where we have nearly everything — rock bands, DJs, whatever — to just get a feeling for the machine and what it can do. The weekend before I flew out to Sydney I also mixed a classical show in Backnang using the unit, and after that I was really looking forward to getting to the Opera House.”

With such promising results, Ralf suggested to Tony that they bring Stephan and his IOSONO machine to the Opera House for this project. He was convinced that by using wave-field synthesis, each musician could be virtually positioned in the pit; creating the illusion of a full orchestra playing underneath the stage as per the traditional format. Moreover, by utilising a large speaker array, they had the potential to place any sound source at any point within the Opera Theatre in three dimensions — giving them plenty of scope for immersive effects. It sounded like exactly what Tony and his team were looking for.

FROM THE SOURCE TO THE SPACE

The audio team ultimately ended up with a total of 56 mics positioned in and around the orchestra to send to the Opera Theatre via a long, sweat-inducing signal-chain. From mobile racks of preamps situated behind the orchestra, the signal is fed via multimode fibre to the Opera House’s Central Control Room (CCR), which then patches the audio through to a Euphonix System5 console in the House’s recording studio. Gain, dynamics and EQ are then applied before the audio is sent to the IOSONO unit via MADI coaxial cables to work its magic; determining each source’s final destination in the theatre space with pinpoint accuracy via a graphic interface. From the IOSONO box the audio heads out through the CCR to two fader buckets in the Opera Theatre via a conversion to Cat5 fibre, which enables balance engineer Tod Deeley to make adjustments to the levels in real-time in the space — a crucial part of the process. These adjustments are then fed back into the IOSONO and out again into the theatre’s MADI to analogue AES before ending up at the d&b amps and a PA system comprising 97 individual speakers. As if that isn’t complicated enough, each of the singers on stage wear a microphone that is fed back to the orchestra and conductor, but their voices are not amplified into the audience because it was strongly felt that would compromise the acoustic qualities of the music.

The ‘bottleneck’ of the system, as Stephan Mauer describes it, is really the speakers: “We can deal with a flexible layout, but if you’re using a good speaker, with good sound itself and in a reasonable arrangement for what you want to do, it helps the result. So there was a strong collaboration between d&b and IOSONO in the process for the club and the classical concert before we arrived in Sydney. There we had five speaker sources across the stage, but here we wanted to increase efforts to get even better spatial results, so we have seven speaker positions across the stage. The algorithm is pretty much adapting to what it’s given, so the machine has to know exactly where speakers are and what they should be used for and what the characteristics are. Then the machine adapts.”

Zuleeg: “We are using d&b T10s for everything full range, as they can be used as line- or point-source speakers. In addition to our seven in the pit representing the orchestra, seven arrays of 6 x T10s are flown in front of the proscenium arc, acting like an artificial canopy, as well as some T10s around the audience areas for enhancing the room acoustics and subtle effects like the choir flying above the audience’s heads. Four Q-Subs are also used to extend the frequency range towards the lower octave.” His choice of speakers came down to one thing: “My argument is that, to make it happen correctly, a loudspeaker has to be as homogeneous as possible. To do the same in all directions into the room, getting the relative angle from the loudspeakers; to be able to reproduce exactly what we want.”

THE MIND-BEND

When it comes to creating a mix, Cray and his team discovered that wave-field synthesis requires a completely different approach that takes some getting used to. As Ralf explains: “It’s really interesting when you’re mixing for the first time on a system like this. There is no left and right master fader anymore, you have to learn to close your eyes and really think about doing what is spatially correct; you just place your instruments. So the very first time it’s really difficult. You start mixing and you think, what’s going on here? It’s not as it used to be.” It may be confusing to begin with but the results are nothing short of liberating. Mauer: “It’s just amazing how easily you get rid of some of the work in the mixing process. You don’t have to think about ‘this speaker should be a little bit louder, this should be a little bit later;’ it’s just done by the algorithm. So it takes lots of crucial work from you and gives you freedom as a mixer.

“The fun thing is that you no longer have only time and frequency domains, but you have this spatial impression — like depth of the venue or the orchestra body — and not just for people who are sitting in the sweet spot. So there are plenty of people who perceive what you mix, it’s not just the five percent on the straight line in the middle of the room. It’s not the same for everybody, but it’s a correct thing. If you imagine it in stereo it would only be stable on one line of seats.”

CRUNCH-TIME

Exactly one week before opening night, and with everything installed and initial troubleshooting completed, it was time to test the system in full-flight with singers in the Opera Theatre. Adding to the pressure was the fact that key personnel in the opera company would be present, and at the end of the day, the conductor and orchestra would sit in the theatre to hear back a recording of an earlier rehearsal. Everyone was encouraged to voice any concerns regarding the sound. Tony admits he had nerves about the day: “My goal for Saturday was to incrementally nibble off the challenges, but the biggest challenge was: does it work when you amplify an orchestra from another venue; to me that was 95% of this job. Everybody would know whether the orchestra sounds right or not, and more than whether they think it does, it’s whether it undermines the integrity of an orchestral performance; that to me was the biggest challenge of this. Is miking an orchestra in a remote location, capturing it and reproducing it, going to diminish the integrity of the experience?”

For Stephan Mauer, there was an initial concern about how digital delay in the signal chain would affect performers. “This is the biggest live-critical venue in a closed hall that we’ve done so far. We’ve done bigger venues with more speakers that are not live-critical, like cinemas or special venues like theme parks, but this is the first time with an orchestra not on the stage, and the singers not being able to monitor the live-sound. We’re doing extensive processing of 56 signals, from 56 virtual sound-sources, plus mixes and effects returns. The processing effort scales up with the number of loudspeakers and the number of feeds that you get into the system, and scales with the complexity you allow the algorithm to use, but it’s very critical to be on time. There’s a visual synchronisation between the conductor, who is hearing the orchestra not reinforced directly in the room, and the singers via mics, and the singers see the conductor via monitors and hear the orchestra via the reinforcement system. I can estimate what the delay of the system will be due to speaker arrangement and the settings you choose; I can come up with a number, but I don’t know yet if it’s working for the artists.”

Imagine the relief when it was proven that the technology not only worked effectively, but actually performed far better than everyone thought it would. In describing his reaction to the test, Zuleeg exclaimed: “This just exceeded my expectations by far; that it works that well. I thought it would still be like an artificial reproduction of the orchestra, but as a matter of fact, if you’re sitting there, it’s much more than that. You can move around, but it maintains the same perspective. Which means when you close your eyes you have the real body of the orchestra in front of you, rather than having to stay in between left and right speakers. What’s really amazing for me is that you get rid of all the masking effects that you get if you have to just melt it all together on two different channels. The violins have their space, celli have their space, and they are there. They are not on top of each other, and for that reason you can exactly identify what they are playing.”

Fears of an unmanageable lag for the performers were also quickly erased.  Stephan found that “the latency is in the range of some metres in the sound radiation, but if you compare this with an actual orchestra with the last desk of the violins being eight metres away from the conductor, we are within a reasonable range. So it just works!”

For maestro Badea it was a huge relief to know that the musicians could feel confident that their performance would be faithfully recreated in the Opera Theatre. In addressing the musicians he said, “It’s sounding fantastic, but the most important thing is that it’s not them that’s making it sound fantastic, it’s you.” Zuleeg was delighted to hear this, as one of his favourite mantras is that “music still happens in front of the microphone.”

For Tony there was another unexpected result: “It’s quite spooky that there are certain frequencies, like in the celli and the brass and some of the higher strings, that when you’re standing on the stage, they sound like they’re down there in the orchestra pit. So even though the system sounds like it requires you to be in the sound-field out the front, there are other parts to it where it’s effected the experience of the performers on the stage. so they get a phantom image of an orchestra underneath them. now i didn’t expect that.”

A SINGER’S PERSPECTIVE

So the sound-team and musicians were satisfied, but it was crucial that the people who have to stand on stage and perform felt comfortable as well. Baritone singer José Carbó sings the role of Fritz in the opera, and having sung in theatres all over the world, has plenty of experience being accompanied by a pit orchestra. As such he was a little hesitant about the idea of phantom musicians emanating from below the stage. “I at first thought, how’s this going to work? The Opera Theatre orchestra pit is a little unusual in that it is covered over. So w use foldback all the time, but still, I was skeptical of the mix between amplified orchestra and natural voice.” Carbó was also concerned about feeling enough of the emotional nature of the music in order to support his performance on stage: “I was interested to see whether we were going to be able to hear enough colour coming out of the instruments, because we need a beautiful, lush blanket to sing upon. That’s what we’re after as singers and why it’s such a pleasure singing with an orchestra; it gives such depth of emotion through its sound.” When it came down to rehearsing with the system up-and-running, however, Carbó happily found that both of these concerns were “addressed masterfully… I was absolutely overawed by the quality of the sound; it was an absolute delight to sing on stage.”

A GAME-CHANGER

For Tony David Cray, this experience was not simply another successful gig, it was the harbinger of exciting things to come in the industry. “There’s the technical aspect of how this comes together, and that’s quite complex, but to me the real lesson that’s coming out of this is: I think audio has changed; what sound mixing is about is actually changing. When i come down here to the recording studio and listen to the orchestra from the multi-track, i feel that’s deficient. whereas what we were doing in the theatre is a whole different game. But it’s exactly the same mics, same orchestra, same setup, the different component is the wave-field synthesis engine, which is recreating what’s real, as opposed to our approximation of it. And that simple fact is such a massive, massive shift.”

Ralf Zuleeg agrees: “A good analogy is that we are now providing a proper canvas to paint the acoustic picture onto the stage. It feels one-dimensional what we’d done so far, but now we’re just extending the limits.”

Stephan Mauer is thrilled with the success of the IOSONO machine, but can’t wait to build on the experience: “What’s great about this is, it’s already working that well. But even now I have ideas, like how we make the process easier and more efficient. The overall number of speakers will ultimately be minimised or at least reduced, and then we can do more, which is really great for me. So it’s really convincing now, and I’m really satisfied, but I still have ideas as to how I, on the application side — together with 15 guys back in Germany working on the algorithm — could change things. It’s going to be cool.”

As Opera Australia proudly claims, their production of Die Tote Stadt is “a celebration of opera in the twenty-first century,” and you can be sure it plans on bringing more innovation to the stage in future shows. So, next time you’re at the theatre, take a good look at the pit and check whether the orchestra is actually there, because judging by the success of this production, creepy organ players with half-face masks may not be the only phantoms at the opera.