Julian Straub

ShapeR: Robust Conditional 3D Shape Generation from Casual Captures 2026-01-16

Yawar Siddiqui, Duncan Frost, Samir Aroudj, Armen Avetisyan, Henry Howard-Jenkins, Daniel DeTone, Pierre Moulon, Qirui Wu, Zhengqin Li, Julian Straub, Richard Newcombe, Jakob Engel

Arxiv 01/2026 project page github video weights

ShapeR generates high-fidelity 3D shapes from casually captured image sequences. It uses SLAM, 3D detection, and vision-language models to extract per-object conditioning and a rectified flow transformer to generate shapes, achieving 2.7x improvement in Chamfer distance over state of the art.

Aria Gen 2 Pilot Dataset 2025-10-17

Chen Kong, James Fort, Aria Kang, Jonathan Wittmer, Simon Green, Tianwei Shen, Yipu Zhao, Cheng Peng, Gustavo Solaira, Andrew Berkovich, Nikhil Raina, Vijay Baiyya, Evgeniy Oleinik, Eric Huang, Fan Zhang, Julian Straub, Mark Schwesinger, Luis Pesqueira, Xiaqing Pan, Jakob Engel, Carl Ren, Mingfei Yan, Richard Newcombe

Arxiv 10/2025 dataset explorer

An egocentric multimodal dataset recorded with Aria Gen 2 glasses covering daily activities including cleaning, cooking, eating, playing, and outdoor walking. Includes raw sensor data and machine perception outputs.

DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos 2025-06-11

Chieh Hubert Lin, Zhaoyang Lv, Songyin Wu, Zhen Xu, Thu Nguyen-Phuoc, Hung-Yu Tseng, Julian Straub, Numair Khan, Lei Xiao, Ming-Hsuan Yang, Yuheng Ren, Richard Newcombe, Zhao Dong, Zhengqin Li

NeurIPS 2025 project page

A feed-forward method for real-time reconstruction of dynamic scenes from monocular video. DGS-LRM predicts deformable 3D Gaussians in a single forward pass, achieving results competitive with optimization-based methods while enabling long-range 3D object tracking.

Sonata: Self-Supervised Learning of Reliable Point Representations 2025-03-20

Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao, Julian Straub,

CVPR 2025 project page github weights

Self-supervised learning of powerful and reliable point cloud representations. Sonata features improve linear probing accuracy by 3x on ScanNet and performance by 2x when given only 1% of training data.

Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling 2025-03-14

Chris Xie, Armen Avetisyan, Henry Howard-Jenkins, Yawar Siddiqui, Julian Straub, Richard Newcombe, Vasileios Balntas, Jakob Engel

ICCV 2025

A system for interactively refining 3D scene layout estimates through human feedback. By framing local corrections as an infilling task on top of SceneScript, the model enables targeted refinements while maintaining global prediction quality.

EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models 2024-09-29

Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, Richard Newcombe

Arxiv 06/2024 github slides talk

The EFM3D benchmark measures progress on egocentric 3D reconstruction and 3D object detection. This accelerates research on a novel class of egocentric foundation models rooted in 3D space. A new model, EVL, establishes the first baseline for the benchmark.

EgoLifter: Open-world 3D Segmentation for Egocentric Perception 2024-03-26

Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, Chris Sweeney

CVPR 2024 github video

We show how to reconstruct and instance segment egocentric Project Aria data using Gaussian Splats.

Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild 2023-10-27

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, Georgia Gkioxari

CVPR 2023 project page github

Omni3D repurposes and combines existing datasets resulting in 234k images annotated with more than 3 million instances and 98 categories. We propose a model, called Cube R-CNN, designed to generalize across camera and scene types with a unified approach.

Orienternet: Visual localization in 2d public maps with neural matching 2023-10-27

Paul-Edouard Sarlin, Daniel DeTone, Tsun-Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulo, Richard Newcombe, Peter Kontschieder, Vasileios Balntas

CVPR 2023 code demo video

We introduce the first deep neural network that can accurately localize an image using the same 2D semantic maps that humans use to orient themselves. OrienterNet leverages free and global maps from OpenStreetMap and is thus more accessible and more efficient than existing approaches.

Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detection 2023-10-27

Yiming Xie, Huaizu Jiang, Georgia Gkioxari, Julian Straub

CVPR 2023 supplementary code

PARQ detects 3D oriented bounding boxes of objects from short posed video sequences.

Project Aria: A New Tool For Egocentric Multi-Modal AI Research 2023-08-24

Jakob Engel, Kiran Somasundaram, Michael Goesele, …, Julian Straub, … Richard Newcombe

Arxiv 08/2023 github

The Project Aria device from my team at Meta Reality Labs Research is an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research. Join Project Aria.

Nerfels: Renderable Neural Codes for Improved Camera Pose Estimation 2022-06-18

Gil Avraham, Julian Straub, Tianwei Shen, Tsun-Yi Yang, Hugo Germain, Chris Sweeney, Vasileios Balntas, David Novotny, Daniel DeTone, Richard Newcombe

Image Matching Workshop CVPR 2022 workshop

We propose to represent a scene as a set of local Nerfs, which we call Nerfels. Nerfels can be used for wide baseline relocalization.

ODAM: Object Detection, Association, and Mapping using Posed RGB Video 2021-10-11

Kejie Li, Daniel DeTone, Yu Fan Steven Chen, Minh Vo, Ian Reid, Hamid Rezatofighi, Chris Sweeney, Julian Straub, Richard Newcombe

ICCV 2021 code

ODAM is trained to detect and track 3D oriented bounding boxes from posed video. We globally optimize a super-quadric-based 3D object representation of the scene.

FroDO: From Detections to 3D Objects 2020-10-27

Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, Richard Newcombe

CVPR 2020

We introduce FroDO, a method for accurate 3D reconstruction of object instances from RGB video that infers their location, pose and shape in a coarse-to-fine manner.

Deep Local Shapes: Learning local sdf priors for detailed 3d reconstruction 2020-08-23

Rohan Chabra, Jan E. Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, Richard Newcombe

ECCV 2020

Deep Local Shapes (DeepLS) is a deep shape representation that enables encoding and reconstruction of high-quality 3D shapes without prohibitive memory requirements.

Habitat: A Platform for Embodied AI Research 2019-10-27

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, Dhruv Batra

ICCV 2019 github

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation.

StereoDRNet: Dilated Residual StereoNet 2019-10-27

Rohan Chabra, Julian Straub, Chris Sweeney, Richard Newcombe, Henry Fuchs

CVPR 2019

StereoDRNet enables the estimation of metrically accurate depth maps enabling high-quality reconstruction by passive stereo video.

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation 2019-07-06

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove

CVPR 2019 github

We introduce neural shape representations in the form of a neural network that can be queried for signed distance values (SDFs).

The Replica Dataset: A Digital Replica of Indoor Spaces 2019-06-13

Julian Straub, Thomas Whelan, Lingni Ma, …, Michael Goesele, Steven Lovegrove, Richard Newcombe

Arxiv 06/2019 github

The Replica dataset consists of 18 high resolution and high dynamic range (HDR) textured reconstructions with semantic class and instance segmentation as well as planar mirror and glass reflectors.

Reconstructing scenes with mirror and glass surfaces 2018-08-01

Thomas Whelan, Michael Goesele, Steven Lovegrove, Julian Straub, Simon Green, Richard Szeliski, Steven Butterfield, Shobhit Verma, Richard Newcombe

SIGGRAPH 2018

We reconstruct mirrors in scenes by detecting a marker on the scanning device. This solves one of the most common failure cases of indoor reconstruction.

Direction-Aware Semi-Dense SLAM 2017-09-18

Julian Straub, Randi Cabezas, John J. Leonard, John W. Fisher III

Arxiv 09/2017

Toward fully integrated probabilistic geometric scene understanding, localization and mapping, we propose the first direction-aware, semi-dense SLAM system.

Nonparametric Directional Perception 2017-05-15

Julian Straub

PhD Thesis 2017

From an indoor scene to large-scale urban environments, a large fraction of man-made surfaces can be described by only a few planes with even fewer different normal directions. This sparsity is evident in the surface normal distributions, which exhibit a small number of concentrated clusters. In this work, I draw a rigorous connection between surface normal distributions and 3D structure, and explore this connection in light of different environmental assumptions to further 3D Perception.

The Manhattan Frame Model -– Manhattan World Inference in the Space of Surface Normals 2017-02-01

Julian Straub, Oren Freifeld, Guy Rosman, John J. Leonard, John W. Fisher III

TPAMI 2017 github

We introduce the Manhattan Frame Model which describes the orthogonal pattern of the Manhattan World in the space of surface normals.

Bayesian Inference with the von-Mises-Fisher Distribution in 3D 2017-01-01

Julian Straub

Writeup 2017

All you want to know about Bayesian inference with von-Mises-Fisher distributions in 3D.

Efficient Global Point Cloud Alignment using Bayesian Nonparametric Mixtures 2016-03-15

Julian Straub, Trevor Campbell, Jonathan P. How, John W. Fisher III

CVPR 2017 github

Use branch-and-bound to globally align two point clouds using nonparametric distribution estimates of their point and normal distributions.

Semantically-Aware Aerial Reconstruction from Multi-Modal Data 2015-12-07

Randi Cabezas, Julian Straub, John W. Fisher III

ICCV 2015

We propose a probabilistic generative model for inferring semantically-informed aerial reconstructions from multi-modal data within a consistent mathematical framework.

Streaming, distributed variational inference for Bayesian nonparametrics 2015-12-05

Trevor Campbell, Julian Straub, John W. Fisher III, Jonathan P. How

NeurIPS 2015 github

How to perform Bayesian nonparametric inference on streaming data in a distributed parallelizable way.

Real-time Manhattan World Rotation Estimation in 3D 2015-09-28

Julian Straub, Nishchal Bhandari, John J. Leonard, John W. Fisher III

IROS 2015 github

We show how to use surface-normals to estimate the global drift-free rotation to a surrounding Manhattan World – a real-time “structure compass”.

A Dirichlet Process Mixture Model for Spherical Data 2015-05-19

Julian Straub, Jason Chang, Oren Freifeld, John W. Fisher III

AISTATS 2015 github

We introduce a nonparametric Dirichlet Process mixture model over the sphere via tangent-space Gaussian distributions. This allows modeling complex distributions on the sphere.

Small-Variance Nonparametric Clustering on the Hypersphere 2015-05-19

Julian Straub, Trevor Campbell, Jonathan P. How, John W. Fisher III

CVPR 2015 github

We show how to derive a fast nonparametric DP-means algorithm, DP-vMF-means, for directional data like surface normals. This allows us to analyze surface normal distributions of depth images.

A Mixture of Manhattan Frames: Beyond the Manhattan World 2014-10-27

Julian Straub, Guy Rosman, Oren Freifeld, John J. Leonard, John W. Fisher III

CVPR 2014 github

We propose a novel probabilistic model that describes the world as a mixture of Manhattan Frames: each frame defines a different orthogonal coordinate system.

Fast Relocalization for Visual Odometry using Binary Features 2013-09-15

Julian Straub, Sebastian Hilsenbeck, Georg Schroth, Robert Huitl, Andreas Möller, Eckehard Steinbach

ICIP 2013 github

We use locality-sensitive hashing to speed up binary feature retrieval for fast camera relocalization.