How Nurulize is building the future of virtual reality
This is the future:
You visit your favorite museum. There's a statue there you really like, an ancient sculpture that is one of a kind, and has always totally inspired you. You go to this museum all the time to look at the statue. It's one of your favorite things in the whole world.
Today, you take a series of pictures of the statue, moving slowly around it to capture different angles. Later, at home, you upload those pictures to be processed.
You put your virtual reality headset. You're in your ideal virtual space, a plush living room based on the home of a Hollywood producer but with elements tweaked, added, and adjusted to fit your style. You can move around this space as you move around in real life. When you sit down in a virtual chair, you sit down in a real chair in the real world that you've mapped so it occupies the same space.
You invite your friends over to hang out in your virtual house. You watch movies and play games together. You head through a door in your house and are suddenly on a set from the latest Jurassic Park film. That's where you keep your drum machine.
You take turns making music on a photo-realistic recreation of an 80s drum machine, tweaking the nobs and flipping the switches with your VR system's touch controls. When you have a beat you think is cool, you switch over to a virtual theremin, making music with the VR equivalent of that most-futuristic of instruments. You combine your tunes together and upload it to your social media for your friends to check out, all without taking off your VR headset.
You leave Jurassic Park and take a look at an empty corner of your virtual living room, a spot you've been saving for something special. It's ready. You open an application inside your VR home and access the model of the museum statue that has been created based off of your real-world photos. Cutting-edge software has produced a virtual reality model and filled in the visual gaps, leaving you with an authentic recreation of your favorite statue you can now add to your ideal virtual space. It looks just like the real thing, and now it's all yours.
If you want to know what virtual reality is going to look like in all of our lives over the next decade, you want to look to the work currently underway from Nurulize.
I recently spent some time in Nurulize's LA offices to check out what they're working on. I looked them up after AMD's Roy Taylor recommended them in a recent interview, identifying their projects as some of the best VR experiences he has personally had. As it turns out, he wasn't kidding.
Scott Metzger, the company's Chief Creative Officer, walked me through a series of demos. Nurulize's main concern these days is its Nu Reality project, which currently consists of an impressively photo-realistic recreation of the home of a Hollywood producer. The home really exists near Joshua Tree outside of Los Angeles, but once Nu Reality is released you'll be able to wander around your own version of the home in the comfort of your living room.
I spent a full hour in the Desert Home setting on the HTC Vive Pre headset, taking a look at all the functionality currently built in to the experience. Starting off, I used the Vive's motion-sensitive controller to define the physical location of a computer desk, the chair in front of it, and a sofa off to the side. Once that was done, certain seats in the virtual environment were mapped to the real-world seats. Once I had worked up enough courage and had done it a few times, I could reliably move around the space and physically sit down in virtual chairs. It was an odd sensation, to say the least.
Created from multiple HDR images
One of the key elements that makes Nu Reality so much more impressive than other VR diversions is the sheer sense of place you experience. It's not surprising, when you learn a bit more about how the digital environment was created:
For creation of desert home, Nurulize used a high end process of photographic capture using 36.3 megapixel Nikon D810 cameras with multi exposure HDR images and lidar point scanning to build a detailed and accurate replica of a real place. Binaural sound was captured to ensure accurate replication of the sonic signature of the location. In total, an area of nearly six acres was captured in high resolution and the recreation has been built with painstaking detail. All the content has been designed to run at high frame rate even at 4K that future proof’s the VR experience.
A lot of what Nurulize does, including their earlier Rise VR tech demo, is develop technology that pushes the limits of VR realism in this way. The company is at the forefront of combining accurate depth models produced by laser scans with multiple HDR images to deliver virtual experiences that approach real-world levels of fidelity. Everything that exists in the Desert Home experience is based on real-world objects, which helps the experience work as a gateway to virtual reality. Flying through space or shooting zombies can be hard to adjust to, but sitting in a chair in a nice house is an experience just about everybody can enjoy with little trouble.
This impressive realism overlays a functionality that, at its core, is pretty simple. Nu Reality isn't a game, or a movie — instead it's a personal VR environment you can customize and populate with the VR equivalent of apps. Here's how Nurulize describes it:
Nurulize has many things planned for release of Desert Home with downloadable content. It’s designed to be a place to relax, work or play with others in a collaborative way, or just sit on the porch and stargaze. Desert home is where you will be able to download and add new experiences, add beautiful art objects, and customize the home to suit your taste. Live feeds of Facebook® and Instagram® will stream into custom picture frames. You’ll be able to watch tv on the screen in the room, or enlarge your monitor to IMAX® size to simulate a drive-in theater. Manufacturers will be able to preview products in VR in a real environment showing them at their best.
Oh the things you can do
Here's what I did during my hour in Desert Home:
- Walked around being flabbergasted by how real it all looked.
- Sat in chairs (in VR and real life simultaneously) and looked at my virtual home some more.
- Sat at a table and made music with a virtual theremin, then did the same thing with a virtual drum machine (which I was told was the same exact model Phil Collins used).
- Leaned in really really close to the drum machine, marveling at the level of detail.
- Sat at a virtual computer on my virtual desk and played Dota 2. As in, I actually played Dota 2 displayed on a virtual computer monitor, with my hands on a real-world keyboard and mouse. I suck at Dota 2 and playing it this way made me even worse, but thankfully the match was only against bots.
- Went into a virtual garage to examine a full-size TurboSquid 3D model of a car in photo-accurate detail, examining it from every angle.
All of these experiences were cool, to varying degrees. The garage experience, for example, could revolutionize the world of car buying if it was adopted by the industry at large, and I discovered a personal interest in drum machines that I never would have realized without this VR opportunity.
Of course some parts of the demo, like the virtual theremin, were only mildly entertaining, but it's not hard to see how they could be much more, for the right person. Playing the VR theremin wasn't particularly exciting to me but, for fans of that esoteric instrument who can't afford several hundred dollars for a real one, it's easy to imagine how a VR theremin app could be a fantastic opportunity, and allow someone to experience something very close to the real thing for a fraction of the cost.
I found it hard to imagine why someone would want to play Dota 2 on a virtual computer monitor and a virtual Razer Black Widow keyboard rather than the real things, but the fact that it was actually happening was impressive, to say the least. And conceivably VR technology like this could make monitors a thing of the past, so maybe my experience was a preview of how we'll all be playing MOBAs in a decade. Nurulize's Scott Metzger offered the idea of a "virtual LAN party" in which your Dota 2 team joins you around a virtual table while you play, which could appeal to some out there nostalgic for the golden age of LAN. According to Metzger, once VR displays are high-res enough, people won't need to buy monitors or TVs anymore, because the virtual equivalents will be cheaper, more flexible, and better in many ways.
Shopping for cars in Jurassic Park
"This is just the first," Metzger said, referring to the Desert Home experience that will kick off Nu Reality, "Once we've finished the house we'll go on to create new environments that can break away from the norms of normal reality to make things a little bit more fun."
"We want to make Nu Reality free," he said when I asked about how the company plans to monetize the experience, "and then we do micro-transactions for items and stuff like that. Anything that goes beyond the stock stuff we provide. For instance, let's say we capture a movie set, and we work out a deal to sell the set. So all the functionality we've programmed, all the tools, you can now use it in a set from your favorite film, and basically live in that for virtual reality."
The opportunities presented by Nu Reality for both companies and consumers are staggering, once you start thinking about it a little bit. In five years it may be common for movies to release VR models of their sets alongside trailers as a way to build up hype. You might be able to watch a new movie trailer inside an entire VR world dedicated to that movie and, potentially, even interact with models of the actors in the film in that virtual space.
What I experienced in the VR garage in Nu Reality is very likely a look at the future of shopping. Imagine a world where you can examine every product imaginable in full visual detail, created in perfect accuracy from CAD data, in the comfort of your own home. You'll sit inside of your dream car in VR. Examine a new refrigerator inside and out before you decide if it's right for you. Rotate an article of clothing in virtual space, checking it out from all angles. These ideas have companies drooling at the possibilities, and consumers will benefit from companies looking to outdo each other with the quality and entertainment value of their VR experiences.
Plus, with Nu Reality combining VR apps and tools and environments, you might be able to sit inside your dream car in the middle of a set from the latest Star Wars movie. So that's a plus, right?
Metzger took me through a variety of demo scenes created through different scanning techniques, many of which were captured using a Lidar scanner. Lidar scanners use lasers to capture depth and dimension information from a surrounding area, and though they're currently professional tools that can be quite expensive, it's likely we'll see prices come down as consumers become more interested in the devices for private use. Like, for example, scanning their actual living rooms in real life, combining that depth data with photographs taken from the same spot as the scan, and generating VR assets and environments through technologies developed by Nurulize and other companies.
The combination of depth scanners and virtual reality will revolutionize the way people experience places within our lifetimes. As Metzger explained, creating accurate scans of an area or object currently only takes a few minutes, and produces a file that can quickly and easily be turned into VR content. What that means in practice is that it will soon be possible to go on vacation, capture depth information and photos of your favorite spots on your trip, and e-mail that VR experience as virtual, explorable postcards before the day is over. Phones in the near future, with technology like Intel's RealSense built into them, will be able to do the depth scanning and photography in a single device, allowing users to photograph statues, pets, trees, or anything else they can imagine and turn that data into VR objects they can use to populate their Nu Reality homes.
As Metzger explained, the Desert Home environment required himself and another developer with years of experience months to create. With the technologies the company is currently developing, people with no professional experience in computer graphics will be able to get "maybe 90% of the way there in terms of quality."
The final part of my demo with Nurulize was a glimpse at Atom View, which is a technology the company is using to create their virtual worlds. Here's how Metzger explained the significance of what Atom View offers for VR development, as he walked me through a VR scene on the surface of the planet Mars:
"For computer graphics, there's real-time, which is games, and there's offline, which is rendering. So when you go to see Star Wars, all the effects and the CG is all generated from 3D renders, which are all offline. So in this Mars scene, the environment you see was created 10 years ago. Everything you see is a point. So if you get close to the ground, I'll turn the filling off. I'm turning off the algorithm we developed to actually display these points and make it look whole..."
The surface of the planet beneath my feet, which had looked as solid and consistent as any other VR experience I have tried, dissolved into a collection of colored points with notable black space in between. The vague outlines and colors of the world were still recognizable, but it was now a super low-resolution version of the environment, built out of nothing but points. Atom View fills in the appropriate visual information between points depending on how close the user is and where they are looking, and allows for VR environments to be built to an incredible level of detail without the rendering demands of traditional methods. If you think of these points as 3D pixels specifically designed for VR, you won't be far off from the truth.
Metzger turned the filling algorithm back on, and continued:
"So this level of detail, you wouldn't be able to do with polygons, because you have to reduce the polygons dramatically to get the performance and detail you're actually seeing inside. If you look over to the sun...this is what's called true HDR. The amount of color information mimics the real world in terms of lighting information that is actually stored in every single point. This scene was rendered in nine minutes. So think of all the existing computer-generated content over the past 30 years that can still be used..."
That last part is the key, really. Atom View provides the tantalizing possibility of a fast and easy way to turn the vast library of existing CG content, from games and films and everything else, into virtual reality assets, complete with depth and the ability to explore your environment. Classic maps from Half-Life 2 could potentially become additional rooms in your Nu Reality home. You could have a CG character like Jar Jar Binks hanging out in your virtual backyard, and then you could throw virtual garbage at him!
The future is amazing. And it's coming sooner than you think.
For more virtual reality news and deals, visit Newegg's VR Central.