Published October 22, 2022 | Version 1.0.0
Dataset Open

GigaDepth: Learning Depth from Structured Light with Branching Neural Networks

  • 1. TU Wien, Vienna, Austria
  • 2. ROR icon TU Wien
  • 3. ROR icon University of Technology Sydney

Description

Structured light-based depth sensors provide accurate depth information independently of the scene appearance by extracting pattern positions from the captured pixel intensities.

Spatial neighborhood encoding, in particular, is a popular structured light approach for off-the-shelf hardware. However, it suffers from the distortion and fragmentation of the projected pattern by the scene's geometry in the vicinity of a pixel. This forces algorithms to find a delicate balance between depth prediction accuracy and robustness to pattern fragmentation or appearance change. While stereo matching provides more robustness at the expense of accuracy, we show that learning to regress a pixel's position within the projected pattern is not only more accurate when combined with classification but can be made equally robust. We propose to split the regression problem into smaller classification sub-problems in a coarse-to-fine manner with the use of a weight-adaptive layer that efficiently implements branching per-pixel Multilayer Perceptrons applied to features extracted by a Convolutional Neural Network.

As our approach requires full supervision, we train our algorithm on a rendered dataset sufficiently close to the real-world domain. On a separately captured real-world dataset, we show that our network outperforms state-of-the-art and is significantly more robust than other regression-based approaches.

Table of contents

  • dataset.zip: Training and validation data, synthetic as well as captured.
     
  • dataset_jpg.zip: The same dataset but with lossy jpg encoding for IR images.
     
  • trained_models.zip: Models trained on the synthetic data.

Files

dataset_gigadepth.zip

Files (608.4 GiB)

Name Size
md5:9804b893c4221fec77f4bf34249091e8
431.4 GiB Preview Download
md5:c853d8cb07e2621057e5ac7c6929c5d5
174.3 GiB Preview Download
md5:77883eea7014d2168e2f73171443f3e1
2.7 GiB Preview Download

Additional details

Dates

Accepted
2022-10-23