Interactive 3D scene reconstruction

FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

From an unsegmented image or casual video, one feed-forward system discovers the objects and estimates their 6-DoF poses, then reconstructs complete textured assets in under a minute.

Hongchi Xia1 Tianhang Cheng1 Wei-Chiu Ma2 Shenlong Wang1

1 University of Illinois Urbana-Champaign 2 Cornell University

Paper Code Models Inference data
31-second overview From unsegmented observations to independently editable 3D scene assets.
Fire3D reconstructs object-compositional, textured indoor scenes from visual observations.
No supplied object masks, boxes, poses, or prompts. Fire3D predicts the scene decomposition and reconstructs complete, textured, independently editable assets, including the room background.

The paper

Abstract

We present FIRE3D, a unified framework that transforms a single RGB image or casual video into interactable 3D scene assets for games and interactive applications in under a minute for up to twelve textured instances including preprocessing. At the core of FIRE3D is a feed-forward inference pipeline that predicts a compositional scene representation from posed RGB-D observations estimated from the RGB capture, including the 6-DoF pose, bounding box, mesh, and texture for detected objects. By modeling the scene as a collection of discrete entities, FIRE3D produces amodal object assets that can be independently edited and used in interactive applications. Our framework requires no test-time optimization, runs substantially faster than the compared reconstruction systems, and provides object-level completeness beyond existing feed-forward 3D approaches. We demonstrate leading detection accuracy, strong geometry reconstruction, and competitive rendering quality on the evaluated benchmarks, with substantial runtime gains.

The missing capability

What other pipelines ask you to provide, Fire3D learns to predict.

Strong 3D methods usually solve one part of the scene problem. Perception systems stop at boxes and masks. Object generators start after an object has been selected. Optimization-based systems recover a scene, but pay a new optimization cost for every capture. Fire3D closes these stages in one learned, feed-forward system.

01 / Joint perception

No supplied object poses or segmentations.

The user supplies an RGB image or casual video. Pi3X, the Pi3 variant used in Fire3D, estimates point maps and camera poses; Fire3D predicts the object count, 3D masks, oriented boxes, and 6-DoF poses without supplied masks, boxes, object poses, or prompts.

02 / Complete output

Complete, independently editable scene assets.

Every predicted entity becomes an amodally complete mesh with PBR appearance in the metric scene. Foreground objects remain independently transformable, while structural surfaces are reconstructed as a background instance.

03 / Scene-scale generation

A scene is processed as a batch of objects.

HC-VAE reduces nominal latent feature capacity by 32×, allowing flow sampling, VAE decoding, and mesh conversion to run across as many as 16 objects in parallel on one GPU, without test-time optimization. Generation uses a fixed 12-step flow-sampling schedule.

Positioning

One system spans perception, completion, appearance, and scene assembly.

Representative methods are grouped by the part of the problem they are designed to solve.

Method family Primary strength Missing or external stage Scene-level outcome
Scene perceptionSceneScript · EFM3D · Boxer Boxes, masks, layouts Complete geometry and material Scene understanding, not editable assets
Selected-object generationSAM 3D Objects · ShapeR · TRELLIS.2 Amodal object reconstruction Object selection, masks, detection, or per-object preprocessing Strong objects; scene decomposition remains external
Per-scene optimizationHoloScene · SimRecon Object-compositional scenes Search, fitting, or iterative refinement for each capture Editable scenes at minutes-to-hours cost
Fire3DSingle image or casual video Joint instance perception and reconstruction No object-level perception input; no per-scene optimization Up to twelve textured instances, including background, in under 60 seconds

Method

A single learned path from observations to interactable 3D scenes.

Fire3D lifts visual evidence into 3D, predicts the scene decomposition, completes each entity in a canonical frame, and places the reconstructed assets back into their predicted metric layout.

Fire3D network architecture from RGB-D observations through scene perception and object reconstruction.
Unified perception and reconstruction. Predicted instance point clouds condition cascaded sparse-structure, shape, and PBR flow models; decoded assets are assembled with the predicted object poses.
Perceive

DINOv3 features are lifted into a shared 3D representation that predicts validity, instance masks, oriented extent, and 6-DoF pose.

Complete

Canonicalized instance points condition structure, shape, and material flows, completing unseen geometry rather than meshing only observed surfaces.

Assemble

Complete textured meshes are transformed back into the world frame as independently editable foreground assets and a static background instance.

Why a full room fits

HC-VAE makes multi-object generation a scene operation.

SC-VAE comes from TRELLIS.2; HC-VAE is introduced by Fire3D. We retain the pretrained SC-VAE representation and add a second, hierarchical compression stage for both shape and material. This changes the practical unit of inference from one selected object to a batch of scene entities.

TRELLIS.2 SC-VAE latent 32³ × 32
Our contribution HC-VAE latent 8³ × 64
Object code
256 KiBtwo float32 grids; sparse metadata excluded
Compression
32×reduction in nominal feature capacity
Inference batch
16 objectson one 80 GB A100
End to end
58.4 s60 frames; 12 instances including background

The total includes 1.05 s for Pi3X on 60 frames, feature extraction, and mesh/texture export. Direct end-to-end timing with batch size 12 also completes within one minute.

The Fire3D hierarchical compression VAE architecture.
Hierarchical compression for shape and material. Fire3D's lightweight sparse 3D encoders compress the SC-VAE fields adopted from TRELLIS.2 into compact latents for batched flow matching and decoding.
Measured trade-off

32× compression with a small reconstruction change

Encoding–decoding quality from Table A6.

Representation Latent Toys4K Imaginarium
CD (cm) ↓F1 ↑PSNR (dB) ↑ CD (cm) ↓F1 ↑PSNR (dB) ↑
SC-VAE only TRELLIS.2 32³ × 32 0.2610.99726.801 0.4070.91921.966
SC-VAE + HC-VAE Fire3D 8³ × 64 0.2690.99126.635 0.4130.91421.665

Interactive 3D Explorer

Inspect the observations, perception, and reconstruction.

Eight held-out scenes are presented in the shared reconstruction coordinate frame. Switch layers to inspect the evidence, predicted instances and boxes, or the final assets.

  1. 01
    Compare representations

    Switch among RGB input, predicted perception, and reconstructed assets.

  2. 02
    Navigate and combine

    Drag to orbit, scroll to zoom, and toggle individual scene layers.

  3. 03
    Edit an instance

    In Reconstruction, click any mesh and transform it in its local frame.

Dataset
View
FloorPlan 312 RGB input point cloud
Loading scene
RGB observations Sampled from the posed input sequence

Evidence

The complete task, tested one claim at a time.

These comparisons cover perception, object completion, end-to-end reconstruction, single-image transfer, and generalization. Real-world quantitative tests cover AEO perception and ShapeR object geometry; rendering comparisons use synthetic data.

Scene decomposition

Discovering objects directly from unsegmented observations

Fire3D leads mAP on all three perception benchmarks and mIoU on iTHOR and Imaginarium. Green boxes are correct predictions, red boxes are incorrect predictions, and orange boxes are missed ground-truth objects.

Controlled reconstruction study

Amodal assets from matched ground-truth perception

All methods receive the same ground-truth instances and poses, isolating reconstruction quality. Fire3D recovers complete geometry and texture from native 3D conditions, while ShapeR is geometry-only and SAM 3D Objects uses each object’s largest-mask input frame. Background is excluded for matched comparisons.

End-to-end scene reconstruction

No masks, boxes, object poses, or known object count

The methods receive the same visual capture. Fire3D performs its own perception and reconstructs textured assets in their predicted scene layout; the EFM3D + ShapeR pipeline composes separate detection and geometry stages. Geometry metrics use matched objects; mAP/mIoU account for missed instances.

Single-image transfer

Automatic scene decomposition and reconstruction from one view

A single RGB image and its Pi3X-estimated point map are sufficient for automatic reconstruction without user clicks or masks. Geometry evaluation aligns the entire scene with one shared scaling, rotation, and translation.

Generalization

Complete foreground and background across diverse scenes

All shown scenes and object assets are held out from training, including examples from training-source datasets. One model reconstructs foreground assets and a background instance across these sources; background quality is shown qualitatively.

Interactive Applications

Reconstructed once. Used as real scene assets.

Independently editable meshes support gaming in Unreal, visual effects in Blender, and robotics demonstrations in IsaacSim/IsaacLab. Systematic evaluation of physical stability and sim-to-real transfer is left to future work.

Dynamic VFXScene-aware effects interact with reconstructed geometry.
RoboticsObject-compositional assets enable embodied interaction.
GamingComplete scene elements remain editable and transformable.

Nearest alternatives

Where the unified design changes the trade-off.

Matched views test Fire3D against optimization-based reconstruction, scene perception, object completion, and a composed detector-to-generator pipeline.

Optimization-based scene reconstruction

Fire3D reconstructs a scene in about one minute; HoloScene requires approximately eight hours of optimization alone, excluding preprocessing. Fire3D improves scene-level CD/F1 and object-level CD/F1; HoloScene achieves higher scene-level NC and PSNR.

View 1 / 16

Open Release

Reproduce the complete pipeline.

The release includes frozen inference protocols, model checkpoints, processed examples for four input domains, camera sampling, and physically based scene rendering.

CodeInstallation and unified inference ModelsPerception, flow, and VAE checkpoints DataReady-to-run example inputs

Citation

Reference Fire3D.

@article{xia2026fire3d,
  title={{FIRE3D}: Feed-forward Interactive 3D Scene Reconstruction Within A Minute},
  author={Xia, Hongchi and Cheng, Tianhang and Ma, Wei-Chiu and Wang, Shenlong},
  journal={arXiv preprint arXiv:2609.08848},
  year={2026},
  url={https://arxiv.org/pdf/2609.08848}
}

Support

Acknowledgment

This project is supported by the NSF Awards #2331878, #2340254, #2312102, #2414227, #2404385, and #2525287. We greatly appreciate the NCSA for providing computing resources through Delta and Delta AI program. We thank the support from the NVIDIA Academic Grant Program for Researchers.