HoloScene: Simulation-Ready Interactive 3D Worlds
from a Single Video

Digitizing the physical world into accurate simulation-ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness.

Given multiple posed images as well as some visual cues (instance masks, monocular geometry priors), we first employ a gradient-based optimization as the initialization. Then we adopt a generative sampling and tree search strategy along the topology of the scene graph to obtain the complete geometry with physical plausibility. Finally, the final fine-tuning over the scene further enhances the realism of the reconstructed scene.

Interactive 3D viewer for reconstructed objects. Drag to rotate, scroll to zoom, and toggle object visibility below.

In our simulation environment, we could also achieve high-quality interactive 3D editing by moving the object Gaussians with its underlying physical mesh geometry. Here we demonstrate this by changing the location and orientation of the interactable chair.

In order to demonstrate the geometry realism and physical plausibility of our reconstructed scene, we select our UIUC Siebel Game Room and print the reconstructed scene. Here we show an overview image, video, and interactions of human-3d printed scene. All objects are movable, and we selected some objects to show the interactions. (Thanks to the printout by Sydney Yu.)