In urban scenes, statistical approach mistakes complex backgrounds for moving objects (top) and fails to remove small moving objects (bottom). On the other hand, Entity-NeRF can reconstruct complex backgrounds and remove small moving objects.
Recent advancements in the study of Neural Radiance Fields (NeRF) for dynamic scenes often involve explicit modeling of scene dynamics. However, this approach faces challenges in modeling scene dynamics in urban environments, where moving objects of various categories and scales are present. In such settings, it becomes crucial to effectively eliminate moving objects to accurately reconstruct static backgrounds.
Our research introduces an innovative method, termed here as Entity-NeRF, which combines the strengths of knowledge-based and statistical strategies. This approach utilizes entity-wise statistics, leveraging entity segmentation and stationary entity classification through thing/stuff segmentation.
To assess our methodology, we created an urban scene dataset masked with moving objects. Our comprehensive experiments demonstrate that Entity-NeRF notably outperforms existing techniques in removing moving objects and reconstructing static urban backgrounds, both quantitatively and qualitatively.
Our Entity-NeRF consists of two approaches
1. Entity-wise Average of Residual Ranks (EARR)
2. Stationary Entity Classification
\(D(\mathbf{r}) = 0\) if Entity-wise Average of Residual Ranks of the entities labeled `thing' in the stationary entity classification is greater than a threshold value \(\mathcal{T}\). The `thing' label for the stationary entity classification is given as \(s(\mathbf{r})=0\) and the `stuff' label as \(s(\mathbf{r})=1\).
This research was partly supported by JST Mirai JPMJMI21H1, JSPS KAKENHI 21H03460, and CISTI SIP.
@article{otonari2024entity,
author = {Otonari, Takashi and Ikehata, Satoshi and Aizawa, Kiyoharu},
title = {Entity-NeRF: Detecting and Removing Moving Entities in Urban Scenes},
journal = {CVPR},
year = {2024},
}