As we all know, in today’s world the text to image generator tools are in demand, and many industries are launching their AI tools that generate images using text.
In this field, open ai launched its own ai image generator tool, Dall-e, which has recently become very popular because its results are impressive. Like Dall-e there are many more ai tools like NightCafe, Artbreeder, and many more.
But these are all image generator ai tools, so this meta raised their limits and created an ai tool which can generate videos using text prompt or some raw images. It can make videos with any type of image. The results are excellent in terms of ai generated videos.
Now, after all this, you might wonder what an ai can do, but you won’t have any idea what an ai can do, so some giant companies like Nvidia and Google have raised the limits of the 3d industry and introduced their new 3d AI generator tool.
Nvidia has launched GET3D, which is trained to generate 3d models using only 2d images with high-fidelity textures and intricate geometric details. These 3D objects are created in the same format used by popular graphics software applications, allowing users to immediately import their shapes into 3D renderers and game engines for further editing.
On the other hand, Google has launched the Dream Fusion AI, which generates 3d models based on 2D images from the generative image model Imagen.
So now as you know about both of the 3d generated ai tools, in this article, we will talk about google’s dream fusion AI.
Dream Fusion AI: Everything You Need to Know
Google unveiled its new 3d generated AI tool named Dream Fusion AI, It is such an extraordinary step taken by google in this dream field. Google combined OpenAI’s image analysis model CLIP with Neural Radiance Fields (NeRF), which allows a neural network to store 3D models for dream fields.
This dream fusion is developed by google research in conjunction with UC Berkeley. DreamFusion generates 3D models from text descriptions like ‘A highly detailed metal sculpture of a squirrel wearing a kimono playing the saxophone.’
How Does Dream Fusion Generate a 3D model?
Dream fusion AI tool works on basically two approaches: Neural Radiance Fields and 2d diffusion. This tool gradually refines a 3d, an initial, random 3D model to match 2D reference images showing the target object from different angles, just like NeRF.
But the NeRF basically uses the actual image, whereas the dream fusion uses the synthetic images generated by a 2D text-to-image model termed imagen.
Google and UC Berkeley propose Score Distillation Sampling (SDS), a way to generate samples from a diffusion model by optimizing a loss function.
“SDS allows us to optimize samples in an arbitrary parameter space, such as a 3D space, as long as we can map back to images differentiably,” they explained.
“SDS alone produces reasonable scene appearance, but DreamFusion adds additional regularizers and optimization strategies to improve geometry. The resulting trained NeRFs are coherent, with high-quality normals, surface geometry, and depth, and are relightable with a Lambertian shading model.” according to dreamfusion3d.github.
Exporting Generated 3D Models for Standard 3D Tools
According to dreamfusion3d.GitHub “Our generated NeRF models can be exported to meshes using the marching cubes algorithm for easy integration into 3D renderers or modelling software.”
Dream Fusion’s online gallery features a variety of models in GLB format that appears to be acceptable for use in an AR project or as basic meshes that might be manually improved for use in higher-detail projects.
Wrapping Up
Google always tries to break the traditional limitation, and now, by developing the dream fusion AI, it has raised the boundaries of the 3d world. This dream fusion AI is an extraordinary step taken by google in the dream fields.
Dream Fusion uses a text-to-image generative model called Imagen to optimize a 3D scene. Now I am really excited about their updates in this dream fusion, and I will always keep you updated through my post.