Figure 1: We propose 3R-GS, a robust method for reconstructing high-quality 3D Gaussians and poses from the MASt3R's imperfect output cameras. Our method outperforms simply joint camera pose optimization along with 3DGS in a large margin.
3D Gaussian Splatting (3DGS) has revolutionized neural rendering with its efficiency and quality, but like many novel view synthesis methods, it heavily depends on accurate camera poses from Structure-from-Motion (SfM) systems. Although recent SfM pipelines have made impressive progress, questions remain about how to further improve both their robust performance in challenging conditions (e.g., textureless scenes) and the precision of camera parameter estimation simultaneously.
We present 3R-GS, a 3D Gaussian Splatting framework that bridges this gap by jointly optimizing 3D Gaussians and camera parameters from large reconstruction priors MASt3R-SfM. We note that naively performing joint 3D Gaussian and camera optimization faces two challenges: the sensitivity to the quality of SfM initialization, and its limited capacity for global optimization, leading to suboptimal reconstruction results.
Our 3R-GS overcomes these issues by incorporating optimized practices, enabling robust scene reconstruction even with imperfect camera registration. Extensive experiments demonstrate that 3R-GS delivers high-quality novel view synthesis and precise camera pose estimation while remaining computationally efficient.
This video demonstrates the novel view synthesis results of our constructed scenes. Our method significantly outperforms naive joint optimization of camera poses and 3DGS in both rendering quality and camera pose estimation. For camera pose estimation results, please refer to Figure 4.
We propose 3R-GS to jointly optimize 3D Gaussians and camera poses from imperfect initial estimates provided by MASt3R-SfM. Our approach addresses two key challenges:
Our solution consists of three key components:
Problem: Gaussian primitives have limited adaptability to poor initialization, as rendering gradients only affect a small local region, preventing primitives from escaping local minima.
Solution: We adopt 3DGS-MCMC, which introduces noise-guided exploration to help Gaussians escape local minima and improve convergence:
This approach eliminates the need for heuristic-based densification and pruning in 3DGS, simplifying joint optimization with camera poses.
Problem: Multiple cameras often share common drift errors, but standard optimization treats them independently, potentially distorting correct local relative poses.
Solution: We employ an MLP-based global pose refiner to predict correlated pose corrections:
This shared MLP captures global relationships across all camera views, enabling more consistent and accurate pose adjustments.
Problem: Standard depth-based geometric constraints for camera optimization require rendering multiple views, which is computationally prohibitive in 3DGS.
Solution: We propose a rendering-free geometric constraint using epipolar distances between image correspondences:
This enables efficient and globally-informed camera optimization without additional rendering overhead.
Our experiments demonstrate that 3R-GS outperforms naive joint optimization approaches in both novel view synthesis quality and camera pose accuracy. Below are some qualitative results:
@misc{huang20253rgsbestpracticeoptimizing,
title={3R-GS: Best Practice in Optimizing Camera Poses Along with 3DGS},
author={Zhisheng Huang and Peng Wang and Jingdong Zhang and Yuan Liu and Xin Li and Wenping Wang},
year={2025},
eprint={2504.04294},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2504.04294},
}