9.2019 - Object tracking in RGBD streams generated by ugmented Reality glasses (Hololens) for a surgery application
Object tracking in RGBD streams generated by Augmented Reality
glasses (Hololens) for a surgery application
Laboratoire : LIFAT - EA 5300
Encadrants : M. Slimane, JY Ramel, C. Proust
Objectives
n a medical application framework (shoulder surgery), this project
aims to set up methods and algorithms capable of recovering,
characterizing and treating separately and / or jointly the video
streams (depth and RGB) coming from AR glasses (Hololens) to then
detect specific elements (gleno for example) allowing then to position
virtual objects (prosthesis) relative to these elements (above for
example) in the reconstructed 3D scene.
To achieve this objective, the first step is, from the streams of the
different cameras, to build a cloud of points (ie a 3D object)
representing the environment around us (area that we want to scan). .
These points must be repositioned in a common reference space so
that they can be reprojected with precision in the HoloLens glasses
with additional virtual objects.
The purpose would be to be able to scan an object of several
centimeters with a precision to the millimeter.
Some methods have already given rise to interesting results (see
github); but they have yet to be investigated in more details:
ScannerObject1: Functional application based on the
MixedRealityToolkit and HoloLensCameraStream plugins to build a
cloud of colored dots. Functional for objects of several tens of
centimeters with insufficient precision.
ScannerObject2: Non-functional UWP (Universal Windows Platform)
application using the Microsoft API to build a point cloud. It uses the
data of the depth camera stream directly, as well as the position and
orientation data of the glasses.
Once this part has been solved, the second step of the study will be
to define and implement an object tracking in the streams or the
reconstructed 3D scene.Outils et langagesGlasses Hololens of
microsoft, the programming software is C# and Unity under visual
studio Github of the projet :
https://github.com/NthnMgns/HoloLens_3D_Reconstitution/wiki
glasses (Hololens) for a surgery application
Laboratoire : LIFAT - EA 5300
Encadrants : M. Slimane, JY Ramel, C. Proust
Objectives
n a medical application framework (shoulder surgery), this project
aims to set up methods and algorithms capable of recovering,
characterizing and treating separately and / or jointly the video
streams (depth and RGB) coming from AR glasses (Hololens) to then
detect specific elements (gleno for example) allowing then to position
virtual objects (prosthesis) relative to these elements (above for
example) in the reconstructed 3D scene.
To achieve this objective, the first step is, from the streams of the
different cameras, to build a cloud of points (ie a 3D object)
representing the environment around us (area that we want to scan). .
These points must be repositioned in a common reference space so
that they can be reprojected with precision in the HoloLens glasses
with additional virtual objects.
The purpose would be to be able to scan an object of several
centimeters with a precision to the millimeter.
Some methods have already given rise to interesting results (see
github); but they have yet to be investigated in more details:
ScannerObject1: Functional application based on the
MixedRealityToolkit and HoloLensCameraStream plugins to build a
cloud of colored dots. Functional for objects of several tens of
centimeters with insufficient precision.
ScannerObject2: Non-functional UWP (Universal Windows Platform)
application using the Microsoft API to build a point cloud. It uses the
data of the depth camera stream directly, as well as the position and
orientation data of the glasses.
Once this part has been solved, the second step of the study will be
to define and implement an object tracking in the streams or the
reconstructed 3D scene.Outils et langagesGlasses Hololens of
microsoft, the programming software is C# and Unity under visual
studio Github of the projet :
https://github.com/NthnMgns/HoloLens_3D_Reconstitution/wiki