Reconstructing 3D models from 2D images has long been a fundamental challenge in computer vision and graphics. While traditional methods rely on multi-view geometry to infer depth and structure, recent advances in neural implicit representations and diffusion models have enabled promising approaches for single-image 3D reconstruction. This blog post explores key methodologies, including NeRF-based techniques, One-2-3-45, Adobe’s Large Reconstruction Model (LRM), and Latent NeRF, evaluating their strengths and limitations in the context of single-image 3D generation.

Point Cloud Generation and NeRF-Based Methods

A foundational step in 3D reconstruction involves generating point clouds from depth maps extracted from 2D images. This point cloud can then be meshed into structured 3D models. One of the most widely adopted approaches is Neural Radiance Fields (NeRF), which trains a neural network to represent continuous 3D scenes from multiple 2D views. However, traditional NeRF methods typically require multiple images to construct an accurate 3D geometry, posing a significant limitation for single-image reconstruction.

A core challenge in single-image NeRF-based reconstructions is the unknown camera pose, leading to inaccurate 3D interpretations. Similar issues appear in diffusion-based image-to-3D models, where structural ambiguities—such as the Janus problem in text-to-3D generation—result in inconsistent back-facing geometries.

One-2-3-45: Multi-View Prediction for Enhanced Reconstruction

To address the limitations of single-image NeRF approaches, One-2-3-45 leverages multi-view prediction to improve 3D reconstruction. This method integrates:

• Zero123, a view-conditioned 2D diffusion model that synthesizes novel perspectives from a single image.

• SDF-based reconstruction, which utilizes these generated views to infer occluded regions and refine 3D shapes.

By generating additional viewpoints before reconstruction, One-2-3-45 mitigates occlusion-related ambiguities and enhances 3D model completeness. However, the reliance on a diffusion-based multi-view approach introduces potential inconsistencies between generated views, occasionally leading to surface artifacts and structural distortions.

Adobe LRM: Transformer-Based Direct Reconstruction

Adobe’s Large Reconstruction Model (LRM) presents an alternative to the multi-view synthesis approach by reconstructing 3D models directly from a single image. Unlike One-2-3-45, LRM:

• Utilizes a transformer-based architecture, bypassing intermediate 2D view generation.

• Leverages a large-scale dataset of 1 million objects, enabling rapid inference.

• Predicts NeRF-based 3D structures in just 5 seconds, significantly reducing computational overhead.

By removing the dependency on per-shape optimization and diffusion-based multi-view generation, LRM achieves smoother, geometrically stable reconstructions with improved consistency. While both methods aim to enhance single-image 3D reconstruction, LRM’s direct-to-3D approach prioritizes speed and accuracy over the iterative refinement process required by One-2-3-45.

Latent NeRF: Towards Interactive AI-Driven 3D Refinement

An emerging direction in 3D reconstruction is Latent NeRF, which moves beyond traditional NeRF and signed distance function (SDF) models by encoding 3D information into a latent space. This approach offers several advantages:

• Flexible shape refinement by allowing modifications in a compressed representation rather than direct pixel manipulation.

• Integration of text-based constraints and geometric priors, enabling more intuitive 3D shaping.

• Interactive artist-guided modeling, where users can iteratively adjust AI-generated 3D structures rather than relying solely on automated outputs.

Latent NeRF represents a step towards making AI-driven 3D reconstruction more dynamic and customizable. Instead of rigidly adhering to photometric accuracy, it allows for creative exploration, where artists can refine structural interpretations based on intent rather than strict geometry.

Conclusion and Future Directions

The landscape of single-image 3D reconstruction is evolving rapidly, with methods like One-2-3-45 and LRM demonstrating different trade-offs between accuracy, speed, and interpretability. While diffusion-based multi-view generation provides richer depth inference, transformer-driven direct reconstruction offers efficiency and stability. Meanwhile, Latent NeRF opens the door for more interactive, artist-driven workflows.

Future research could explore hybrid approaches that combine diffusion-based multi-view synthesis with transformer-based direct reconstruction. Additionally, integrating Latent NeRF into artistic critique and design workflows could bridge the gap between automated reconstruction and human creativity. As these methods continue to develop, they promise to expand the possibilities for AI-driven 3D modeling, from industrial applications to interactive creative tools.

References

1. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.” ECCV 2020.

2. Liu, S., et al. (2023). “One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization.” arXiv preprint arXiv:2306.16933.

3. Adobe Research (2023). “Large Reconstruction Model (LRM): Scalable Single-Image 3D Reconstruction via Transformers.” Adobe Tech Blog.

4. Poole, B., et al. (2022). “DreamFusion: Text-to-3D using 2D Diffusion.” arXiv preprint arXiv:2209.14988.