Interactive medical record

AR journey into the human body

Virtual and augmented reality are already being used in many medical fields. However, this is mostly done to train highly qualified specialist staff. But why not use the technology to benefit patients directly? This is exactly what we did as part of the cross-industry joint project MPM – Models for Personalized Medicine. In this context, we worked on the sub-project: AR-supported assistance for patient information and education.

The result is an iPad application that perfectly illustrates informational discussions in the run-up to an operation and can help take away the fear of such an intervention.

A sympathetic ear for patients

The aim of this project is to improve communication between medical staff and patients in traditional information sessions. To this end, we would like to integrate digital aids into the patient consultation. This is by no means intended to replace the personal dialog, but rather, as augmented reality already bears in its name, to expand reality through the additional use of media and thus create added value in terms of clarity.

Particularly in the context of complex diseases such as cancer, education is becoming increasingly important and can increase patient motivation during treatment. Apart from this, comprehensive information is necessary for legal reasons alone.

iPad instead of clipboard

Currently, educational talks are usually conducted with the help of printed illustrations and manual drawings. We replaced the clipboard and pen with two iPads and this is how it works:

The healthcare professional can create a customized “tour” of a 3D model of the human body in advance. Factors such as the level of detail, perspective or organs shown can be arranged in any order. Markers and notes can also be placed. External media sources such as videos and voice recordings can also be integrated for even better visualization. This allows the treating person to guide the patient through a well-structured presentation during the anamnesis consultation. It is also possible to add live colored markers and freehand drawings.

Patients are given their own tablet during the consultation. They can follow the “tour” on this as an observer. Thanks to their own iPad, they always have a clear view and everything is right in front of them. They can also determine their own viewing angle.

An eye for ENT

Through close cooperation with the head of an ENT ward at Leipzig University Hospital, Dr. med. Matthäus Stöhr, we have focused on the head, shoulders and upper body in order to depict the ENT area in detail. Above all, the aim is to depict surgical options relating to the diagnosis of laryngeal cancer. These are quite complex and therefore an ideal use case for our application.

What we want to see and what we don’t

As the focus is on clarity, the medically correct model is clearly the focus of the application. Nevertheless, there are many pitfalls to consider when it comes to the level of detail.

For example, we have deliberately refrained from showing animations of surgical procedures, tools used or similar. Accordingly, no virtual blood flows here! We want to provide information and reduce fears – not have the opposite effect.

A clear case of too much information. For this reason, in consultation with medical specialists, we have also refrained from depicting disease-related anomalies.

However, to give those affected a good insight into the procedures, we provide a digital version of the pen that was previously used to outline the procedure on paper. We provide several drawing options with different colors for this purpose. These can be entered in advance or added to live during the consultation.

In this way, we manage the balancing act between a better idea of an upcoming procedure than is possible with an illustration on paper. On the other hand, we deliberately do not exploit the full potential of virtual experiences.

Thinking outside the box

What we particularly appreciate about funding projects like this is the opportunity to work on a project together with universities and companies from other sectors. In this case, it was the Innovation Center for Computer-Assisted Surgery (ICCAS) at the University of Leipzig as the contact partner for all medical topics and Effigos AG, which specializes in the creation of 3D models for the healthcare sector.

The project was funded by the Federal Ministry of Education and Research and the Centre for Innovation Competence – Joint Project in the course of the MPM – Models for Personalized Medicine.

Digital medical records: how we illustrate the patient consultation