Dynamic Depth-Supervised NeRF for Multi-view RGB-D Operating Room Videos

Beerend G.A. Gerats*, Jelmer M. Wolterink, Ivo A.M.J. Broeders

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

The operating room (OR) is an environment of interest for the development of sensing systems, enabling the detection of people, objects, and their semantic relations. Due to frequent occlusions in the OR, these systems often rely on input from multiple cameras. While increasing the number of cameras generally increases algorithm performance, there are hard limitations to the number and locations of cameras in the OR. Neural Radiance Fields (NeRF) can be used to render synthetic views from arbitrary camera positions, virtually enlarging the number of cameras in the dataset. In this work, we explore the use of NeRF for view synthesis of dynamic scenes in the OR, and we show that regularisation with depth supervision from RGB-D sensor data results in higher image quality. We optimise a dynamic depth-supervised NeRF with up to six synchronised cameras that capture the surgical field in five distinct phases before and during a knee replacement surgery. We qualitatively inspect views rendered by a virtual camera that moves 180 around the surgical field at differing time values. Quantitatively, we evaluate view synthesis from an unseen camera position in terms of PSNR, SSIM and LPIPS for the colour channels and in MAE and error percentage for the estimated depth. We find that NeRFs can be used to generate geometrically consistent views, also from interpolated camera positions and at interpolated time intervals. Views are generated from an unseen camera pose with an average PSNR of 18.2 and a depth estimation error of 2.0%. Our results show the potential of a dynamic NeRF for view synthesis in the OR and stress the relevance of depth supervision in a clinical setting.

Original languageEnglish
Title of host publicationPredictive Intelligence in Medicine
Subtitle of host publication6th International Workshop, PRIME 2023, Held in Conjunction with MICCAI 2023, Proceedings
EditorsIslem Rekik, Ehsan Adeli, Sang Hyun Park, Celia Cintas, Ghada Zamzmi
Place of PublicationCham, Switzerland
PublisherSpringer
Pages218-230
Number of pages13
ISBN (Electronic)978-3-031-46005-0
ISBN (Print)978-3-031-46004-3
DOIs
Publication statusPublished - 2023
Event6th International Workshop on PRedictive Intelligence In MEdicine, PRIME 2023 - Vancouver, Canada
Duration: 8 Oct 20238 Oct 2023
Conference number: 6

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer
Volume14277
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference6th International Workshop on PRedictive Intelligence In MEdicine, PRIME 2023
Abbreviated titlePRIME
Country/TerritoryCanada
CityVancouver
Period8/10/238/10/23

Keywords

  • Neural radiance fields
  • Operating room videos
  • RGB-D imaging
  • 2023 OA procedure

Fingerprint

Dive into the research topics of 'Dynamic Depth-Supervised NeRF for Multi-view RGB-D Operating Room Videos'. Together they form a unique fingerprint.

Cite this