As the need for efficient upskilling and reskilling of healthcare personnel is further dictated by the extended pandemic, eXtended Reality (XR) can be utilised to improve and extend traditional educational tools and approaches [1,2,3]. To this end, it is essential to identify both the opportunities and the challenges posed to achieve a major evolutionary step and effectively impact the landscape of medical training.

Figure 1: Realistic VR medical surgery environment, created by MAGES

Virtual Reality (VR) and Augmented Reality (AR) training applications must provide high-fidelity, immersive experiences within shared, collaborative, virtual environments. In this context, real-time rigged and soft-body 3D object transformations, deformations and rendering require a significant amount of local processing power. Furthermore, the synchronised data transmission to all users in the same session demands effective use of the network capabilities, especially in 5G infrastructures. Low latency is essential to maintain a high quality of experience (QoE) and immersion, and achieving it is not straightforward, especially when the users are placed in diverse geolocations. Mobility restrictions posed by the state-of-the-art methods, which involve a tethered head-mounted display (HMD) and a local high-spec computer, can be overcome by employing standalone, untethered HMDs. However, despite recent breakthroughs in the respective hardware, the need for high-end GPUs and long-lasting battery autonomy still remains one of the main bottlenecks in realising untethered, highly immersive, real-time applications.

MAGES – A novel approach

Aiming to revolutionise existing medical approaches, ORamaVR introduced MAGES [4,5], the world’s first hyper-realistic VR-based software platform for enhanced surgical training and assessment (see Figure 1).

With MAGES, trainees can perform immersive surgery simulations in a risk-free environment to improve their performances. Built on top of modern game engines (Unity3D & Unreal Engine), its novel Geometric Algebra based, inTerpolation Engine (GATE) may perform virtual character animation by sharing fewer key-frames across the network [6], thus minimising broadcasted data. Its multi-player engine allows a large number of remote concurrent users by adopting a networking functionality beyond the standard client-server model, where a relay server transfers the current state of deformable objects.

Figure 2: Use case conceptual architecture – Exploiting the available, open-source, render streaming implementation based on WebRTC

Embracing 5G – The CHARITY project

To alleviate the imposed constraints on the untethered HMDs, namely GPU processing power, battery life, and mobility, ORamaVR is currently scaling up its product to edge/cloud resources, to reduce the dependency on local resource-demanding HMD technology assets. This approach, along with a proper 5G network utilisation for minimal latency and maximal bandwidth image streaming, will ensure optimal QoE, even for users using low-spec HMDs.

ORamaVR is participating as a use case partner in Horizon 2020 project CHARITY [7] and aims to exploit different data services and resources provided within the project’s cloud-edge compute continuum. The main objective of this use case is to enhance MAGES edge-based components within 5G networks, while impacting CHARITY’s architecture design [8] to accommodate similar applications. The use case design involves three main aspects: i) computation offloading to nearby edge resources, leaving the mobile device responsible for UI, input/output, thus streaming only necessary content to client HMDs; ii) provide a relay server functionality based on an open-source networking API, hosted in the cloud to broadcast messages to remote clients, thus ensuring low latency; and iii) run-time adaptation and dynamic optimization of GATE, based on the network characteristics.

Current research activities focus on altering the Unity3D workflow by dissecting the physics engine from the main application. ORamaVR aims to influence and adopt CHARITY’s service-oriented architecture with two loosely coupled microservices, deployed in nearby edge resources, responsible for i) maintaining the game logic, rendering, encoding and managing the image transfer to the HMD (see Figure 2), and ii) conducting physics engine calculations of soft-body deformations.

Conclusion

MAGES, an XR medical training cloud-native application, aims to revolutionise the medical training landscape. To this end, and in order to effectively accommodate a large number of concurrent users from diverse geolocations, a proper exploitation of 5G networking capabilities is required. To achieve this and ultimately deliver optimal QoE for its end users, ORamaVR develops the necessary microservices, which will be used to both assess and influence the design of CHARITY’s architecture, to properly accommodate XR applications of this magnitude.

References

[1] Birrenbach, T., Zbinden, J., Papagiannakis, G., Exadaktylos, A.K., Müller, M., Hautz, W.E. and Sauter, T.C., 2021. Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial. JMIR Serious Games, 9(4), p.e29586.

[2] Zikas, P., Kamarianakis, M., Kartsonaki, I., Lydatakis, N., Kateros, S., Kentros, M., Geronikolakis, E., Evangelou, G., Apostolou, A., Catilo, P.A.A. and Papagiannakis, G., 2021. Covid-19-VR Strikes Back: innovative medical VR training. In ACM SIGGRAPH 2021 Immersive Pavilion (pp. 1-2).

[3] Zikas, P., Kateros, S., Lydatakis, N., Kentros, M., Geronikolakis, E., Kamarianakis, M., Evangelou, G., Kartsonaki, I., Apostolou, A., Birrenbach, T. and Exadaktylos, A.K., Virtual reality medical training for Covid-19 swab testing and proper handling of personal protective equipment: development and usability. Frontiers in Virtual Reality, p.175.

[4] Papagiannakis, G., Zikas, P., Lydatakis, N., Kateros, S., Kentros, M., Geronikolakis, E., Kamarianakis, M., Kartsonaki, I. and Evangelou, G., 2020. MAGES 3.0: Tying the knot of medical VR. In ACM SIGGRAPH 2020 Immersive Pavilion (pp. 1-2).

[5] Papagiannakis, G., Lydatakis, N., Kateros, S., Georgiou, S. and Zikas, P., 2018. Transforming medical education and training with VR using MAGES. In: SIGGRAPH Asia 2018 Posters (pp. 1-2).

[6] Kamarianakis, M., Lydatakis, N. and Papagiannakis, G., 2021, September. Never ‘Drop the Ball’in the Operating Room: An Efficient Hand-Based VR HMD Controller Interpolation Algorithm, for Collaborative, Networked Virtual Environments. In Computer Graphics International Conference (pp. 694-704). Springer, Cham.

[7] Website of Horizon 2020 project CHARITY – https://charity-project.eu

[8] Makris A., Protopsaltis A., et al. (2021). Cloud for Holography and Augmented Reality. 2021 IEEE 10th International Conference on Cloud Networking (CloudNet), 2021, pp. 118-126, doi: 10.1109/CloudNet53349.2021.9657125.