NEP helps its clients develop and deliver standout live broadcast events. With combined creative and IT expertise, the company delivers services that transform the way global video entertainment is created, managed and distributed. Over the past two years, NEP has built a dedicated Unreal Engine team to service growing client demands for real-time graphics in broadcast scenarios. We spoke with NEP Senior Unreal Engineer Roel Bartstra to get a closer look at their work. 

When did NEP The Netherlands start using Unreal Engine?NEP’s first project in Unreal Engine was a couple of years ago, before I joined the company. The team started working in Unreal to build an augmented reality-enhanced virtual set for the broadcast of a local game show called “I Bet That I Can Do It.” After that project, NEP decided to invest more heavily in hardware, software, people, and R&D. Can you describe what augmented reality in broadcast production means?We add value by creating things that are physically impossible, for example, in scenarios where there’s limited space or the desired studio is booked. Our virtual set customers are typically broadcasting from a green screen set, which are often very small spaces. Sometimes there is a call for a virtual set extension, where there is a physically-built set or set pieces in the foreground, and augmented graphics and props are placed behind the camera. Other scenarios have no physical props and everything behind and in front of the talent is computer graphics. In production, this is referred to as ‘augmented reality’ because the talent can interact with these dynamically generated real-time graphics, and the magic bringing these images together is happening in Unreal Engine, often during a live broadcast. We can build data-driven graphics into these scenarios as well, updating content related to news and sports stats in real time. This was heavily used in the Winter Olympics broadcast by Eurosport, and also by Liberty Global’s Ziggo Sport as part of their Formula 1 coverage. We also have a client preparing to launch a show with a virtual floor, creating a look that would not be achievable in a physical set environment.

 Are you providing services for hire for other production companies, or are you doing the integration for existing production/graphics teams?It depends on the client. We may see that it’s an addition to our multicam operation, but the end-to-end managed graphics services can be delivered separately as well. Sometimes we are involved in the design phase, and for other clients we execute on an existing design and figure out how to make them feasible. Many of these clients are accustomed to providing a raytraced image and getting an idea of how a show will look, in those cases, we work on making that possible in real time in UE4. Why did NEP choose to invest in AR?It was Ralf van Vegten, Managing Director of NEP The Netherlands who really believed that AR would be the future of dynamic virtual set production in broadcast. My background is in the gaming industry, and I started using Unreal Engine fifteen years ago when I did level design for games and modding in Unreal Tournament. I joined NEP almost two years ago to expand the Unreal Engine team, and integrate gaming industry knowhow to meet the current needs of broadcasting. In the early days, there was a level of technical expertise and custom development required that made this very expensive, but NEP has invested heavily in people and technology to make use of AR in broadcast production much more cost effective. We are continuing to revise and improve our methods, and research what aspects of production we can replace with AR, and make the process more practical. With Unreal Engine, the graphics are very photorealistic, so we really can replace entire studio builds with AR as opposed to the build-out of a physical set. Clients have been really happy with the results so far. We started out with one AR show a month, and now we have four or five running simultaneously at any given time, with one or two going live every week.

 Why do you maintain a dedicated Unreal Engine team?Unreal Engine allows us to produce photo-real graphics at high frame rates, which is important as we’re broadcasting at 50 frames per second. Other game engine technologies we tested weren’t able to achieve that level of image fidelity with real-time performance. UE also has an insane number of pre-built tools that are very solid, and make it easy to get started pretty quickly. It is also well integrated with tools from our partners at Zero Density, and both are working together to give us more creative freedom to do even more in the engine ourselves. How do broadcasters manage the investment required in turning part of their offices into virtual studios?It’s all about explaining the added value: AR makes literally everything you can think of possible – which is far more (cost) efficient than using different sets, props et cetera. For example, for Eurosport they took a small 5 X 5 meter break room and turned it into a small green screen stage. By adding skylights into the ceiling they ensured that enough light could come into the space and the costs were minimal. At NEP we are also working on creating a green box studio so that people can come in and rent a fully operational virtual studio setup (pre-configured with all necessary hardware and software) for one-off shows at several of our existing studios in the Netherlands.

What are the key pieces of gear and software involved in these build-outs?This varies per client. For many clients we provide all of the hardware and software in our studios, along with AR camera operators and graphics administrators to control what’s being shown on virtual screens built into the sets. We can also train our clients’ internal teams to setup and operate similar systems once we’ve handed off a graphics package that lives in Unreal Engine. Most importantly, our people make the difference. NEP The Netherlands is a very people-driven organization. Our team is continuously trained on new technologies, workflows and specializations. We stimulate growth, innovation and ambition, because we believe that the highest level of knowledge is needed in the rapidly changing media industry. Is your UE team made up more of artists or technologists?It’s a mix. We have a few 3D artists that are designers, but they come from the game industry. We also have some technical artists who do a bit of modeling, but are mostly creating Blueprint scripts and custom materials, doing compositing inside of Reality - all of the processes on the tech side of implementing a model in a real-time virtual set environment.

 What’s the average number of assets you’re building for clients?It depends on the nature of the show, so for smaller news broadcasts in a small studio, it may be around 10-50 elements, but for something like Formula 1, we’re producing around 100 assets. Those are generally combined with pre-existing assets created by their in-house teams so there are probably closer to 1,000 for that show. Are there different considerations when building digital assets for virtual sets than other types of photo-real CG productions?When we create assets digitally, we want them to look as lifelike as possible. In order to make the sets look ‘real’ we actually have to build models with slight flaws such as seams between wood panels and misaligned edges, otherwise they are too perfect to be convincing.  Are broadcasters embracing Unreal Engine to save time and money or because they want more flexibility in the visual look of their shows?Flexibility is something that we’re working on for sure, but generally this is deployed for cost and time-savings. This is especially the case if a show uses a lot of LEDs on set. LEDs are practically free to create digitally, so compared to a physical LED set, building virtual screens is much less expensive. Does using game engines have the chance to change the way things are done in broadcast?Yes, Unreal Engine is on its way to becoming a true game changer in virtual production. It’s no longer necessary to build physical sets, as it’s now possible to generate, iterate and revise virtual set design and assets dynamically and in real time. Unreal Engine is a transformative technology for news broadcasts, sports commentary, game shows and any type of live or produced production that benefits from real time data-driven computer graphics, whether stylized or photo-real.