NVIDIA’s AI turns a series of 2D photos into 3D scenes in the blink of an eye
When the first instant image was taken 75 years ago with a Polaroid camera, it was considered a breakthrough as humans were able to quickly capture the three-dimensional (3D) world into a single image. realistic two-dimensional (2D) image. Today, artificial intelligence (AI) researchers are doing the opposite: turning a series of still images into a digital 3D scene in the blink of an eye.
Known as “inverse rendering,” this process uses AI to calculate the movement of light in the real world, allowing researchers to reconstruct a 3D scene from a series of 2D images. Shoot at different angles. The NVIDIA research team has developed a technique that accomplishes this task almost instantaneously – making it one of the world’s first models to operate through a combination of neural network training and training. Super fast and high speed rendering.
NVIDIA has applied this technique to a fairly popular new technology called neural light field, or NeRF. The result – Instant NeRF – is the fastest NeRF technique to date, which in some cases produces products up to 1,000 times faster than conventional. This model takes just a few seconds to learn from a few dozen still images – plus data about the angles at which the photos were taken – and can then render a 3D scene in less than 10 milliseconds.
“If traditional 3D images, like meshes of polygons, have the properties of vector images, then NeRFs are like bitmaps: they capture the way light falls on an object or in a scene.” – according to David Luebke, vice president of graphics research at NVIDIA.In that sense, Instant NeRF can be just as important for 3D photography, just as digital cameras and JPEG compression are for 2D photography – it dramatically improves speed, ease, and range. micro-access to 3D chụp capture and sharing“
Shown in a session at NVIDIA GTC last week, Instant NeRF showed it could be used to create avatars or scenes for the virtual world, to 3D render video conference attendees and the environment. their own, or to reconstruct scenes for 3D digital maps.
And to pay homage to the early days of Polaroid photography, the NVIDIA team has recreated a classic shot of photographer Andy Warhol taking snapshots, turning it into a 3D scene using Instant NeRF .
What is NRF?
NeRF uses neural networks to render and render realistic 3D scenes based on a series of 2D images provided by the user.
Collect data so that NeRF “learns” like you’re a red carpet photographer, trying to capture a certain celebrity’s outfit from any angle – the neural network requires several dozen shots from multiple locations around the scene, as well as the camera position of each of those shots.
In a scene that includes people or other moving elements, the faster the photo is taken, the better the results will be. If there is too much movement during 2D photography, the AI-generated 3D scene will be blurred.
From those “materials”, NeRF essentially fills in the gaps, training a small neural network to reconstruct the scene by predicting the color of the light shining in any direction, from any direction. any point in 3D space. This technique can even overcome some limitations – for example, when objects in some photos are blocked by obstacles such as pillars in others.
Get a 1,000x speedup with Instant NeRF
While estimating the depth and appearance of an object based on incomplete information is a natural human skill, AI has quite a bit of trouble doing the same.
Creating a 3D scene with traditional methods requires many hours, or even longer, depending on the complexity and resolution of the image. Using AI will help speed up this process. Early NeRF models produced sharp, error-free scenes in minutes, but training them still took hours.
However, Instant NeRF has a much shorter rendering time. It is based on a technique developed by NVIDIA, called “multi-resolution hash grid encoding”, which is optimized to run smoothly on its GPUs. Using a new input encoding method, the researchers were able to achieve high-quality results using a small, high-speed neural network.
This model was developed using the NVIDIA CUDA Toolkit and the Tiny CUDA Neural Networks library. Because it’s a lightweight neural network, it can be trained and run on a single NVIDIA GPU – and is fastest on cards with NVIDIA Tensor Cores.
The technology can be used to train robots and autonomous vehicles, helping them understand the size and shape of real-world objects by taking 2D photos or videos of the objects. It can also be used in architecture and entertainment to quickly create digital images of real environments, allowing creators to further edit and develop.
In addition to NeRF, NVIDIA researchers are learning how to apply this input encoding technique to speed up many other AI tasks, including reinforcement learning, language compilation, and targeted deep learning algorithms. mass destination.
Reference: NVIDIA
Nvidia “shows off” customers are paying an extra 300 USD to upgrade graphics cards
at Blogtuan.info – Source: genk.vn – Read the original article here
