Development of Augmented Reality and Virtual Reality Application for Dental Education

Abstract

The advent of cheaper and more powerful mobile phones and tablets has made augmented reality (AR) and virtual reality (VR) much more accessible in recent years. These technologies have been used in dental education, to provide virtual models, but there have been problems with the manipulation of the models through the virtual interface. This paper describes an application that can be used with commonly available devices for both AR and VR, and which allows dental students to manipulate the virtual model by manipulating a real-world object.

Introduction

The use of three-dimensional virtual models has proven to be a valuable educational tool in dentistry.¹ However, the most recent advances have allowed educators to enhance students’ dental education further through augmented reality (AR) and virtual reality (VR).² AR uses computer-generated virtual images and/or three-dimensional models to enhance objects in the real world, while still allowing and, in fact, encouraging interaction with those objects. On the other hand, VR completely immerses the user into an artificial virtual environment. AR and VR typically achieve these simulations through the use of a headset that is connected to a computer or mobile device, such as a smartphone or tablet. An individual using an AR headset device is able to see their real-world surroundings overlaid with computer generated images, videos and/or three-dimensional virtual models. AR may also be experienced without the use of a headset device, through the camera of any handheld device, such as a smartphone or tablet. The screen on the handheld device utilizes the built-in camera to display the user’s surroundings overlaid with the selected computer-generated content.

AR and VR have been in development since the late 1950s. The historical use of AR and VR has been largely limited to the videogame industry, but recent years have seen AR and VR make their way into many broader applications. The widespread adoption of smartphones and tablets, along with the exponential increase in their processing power, have made it possible to make AR and VR widely available. AR and VR headset devices such as the Google Cardboard (Google LLC, Mountain View, CA, USA) provide accessible AR and VR experiences at very low costs.³ Moreover, evidence suggests that AR and VR have the potential to enhance students’ learning processes and increase their interest across in many disciplines.⁴ A recent review of the literature identified 87 articles evaluating the use of AR and VR in anatomical education.⁵ Results show that 65% and 95% of the articles for AR and VR respectively concerned the use of AR and VR in anatomical education.⁵ Of course, there has also been a growing interest in the use of AR/VR in the education of health professionals.⁶

The purpose of this article is to describe how AR and VR can be used to enhance accessibility and cost-effectiveness in dental education.

Current Status of Dental AR and VR Applications

Various forms of AR and VR have made appearances in dentistry.⁷ While they do offer a three-dimensional display, most do so on a screen, and do not offer truly immersive AR and VR environment. However, some studies did report true AR and VR.

One process used augmented reality as an adjunct, by means of a desktop computer screen and the supplied webcam.⁸ Another approach assessed the use of a headset device linked to a computer that was utilized to view three-dimensional virtual tooth models in VR.⁹ The results show high acceptance by preclinical students.⁹ However, current AR and VR systems for dentistry have some drawbacks that hamper their adoption in dental education. The first problem is difficulty in manipulating the three-dimensional virtual models. In one study that assessed the use of AR in dental anatomy education, the difficulty in controlling the three-dimensional models led to unfavorable student survey results.¹⁰ The excess effort required to control the models was attributed to inaccessible virtual interfaces. The other disadvantage is the need for bulky, stationary and expensive hardware. Most published AR and VR experiences require a high-performance computer as well as a VR headset device, both of which are associated with hefty price tags, and limited accessibility due to their limited portability.

Development of a Dental AR/VR Application with headset and hand-held versions

In order to overcome the drawbacks mentioned above, a dental AR application that runs on regular smartphones and tablets, requiring no expensive or bulky hardware, was developed by the authors. This application employs a real-world object called a “Target” or “Anchor” to control the virtual three-dimensional models. In other words, the user is able to control and manipulate the virtual three-dimensional model by moving a physical real-world object. The target may be an image on paper or on a screen. Two versions of the application were developed: The first version involves generic AR/VR headset de vices that are made to house a smartphone. (Fig. 1 upper) The second version requires no headset device whatsoever, and displays three-dimensional virtual models in AR through the screen of a smartphone or tablet. (Fig. 1 lower).

Fig. 1

The following section describes how each version of the AR/VR application can be used, and how the three-dimensional virtual models are manipulated.

Headset version (Fig. 2)

1. The target image is displayed on a screen or printed on a piece of paper.
2. The mobile device is placed in the AR/VR headset. For this article, the MERGE generic AR/VR educational headset (Merge Labs Inc., San Antonio, TX, USA) was used.
3. The headset is placed on the user’s head, and the user holds the target image in their field of view. The application instantly recognizes the target image, and displays the three-dimensional virtual model on top of the target image.
4. The user can then move the virtual three-dimensional model by moving the physical target image.

Fig. 2

Hand-held version (Fig. 3)

1. The target image is displayed on a screen or printed on paper.
2. The target image is held in the field of view of the smartphone camera. The application instantly recognizes the target image and displays the three-dimensional virtual image on top of the target image.
3. The user can then move the target image to move the virtual three-dimensional model.

Fig. 3

Discussion

The three-dimensional virtual models used in both versions were developed using virtual scans of natural teeth, detailed and textured using the open-source three-dimensional content creation software Blender (Blender Foundation, Amsterdam, Netherlands). The target images were created using Microsoft PowerPoint (Microsoft Corporation, Redmond, WA, USA).

The headset version of the application was developed using the Unity game engine (Unity Technologies, San Francisco, CA, USA), which incorporates real-time image tracking of customizable target images. Stereoscopic vision, which describes the way in which the brain generates a three-dimensional perception on the basis of visual inputs, was essential to the development of this version. In AR and VR devices, stereoscopy is typically accomplished through algorithms that simulate geometry depth, eye convergence, and individual eye viewing angles. Stereoscopic display on the headset version of the application was achieved by incorporating the Vuforia software development kit (PTC Inc, Boston, MA, USA) into the Unity game engine. When used with AR/VR headsets, the user experiences binocular vision of the real world superimposed with AR virtual three-dimensional models that are properly scaled with the target image in the real world. The application was programmed to render the virtual three-dimensional models on top of the image targets, and track them within the field of view of the smartphone’s camera.

The handheld version of the application was developed using the software development platform XCode (Apple Inc, Cupertino, CA, USA). This application also incorporates real-time tracking of target images, and the application was programmed using the SWIFT coding language so that each three-dimensional virtual model is rendered when the corresponding target image is detected by the mobile device’s camera. This version of the application can render multiple targets simultaneously and with a wider range of movement.

However, it is important to note that this technology is evolving quickly. Evidence from the US patent office points towards a revolution in AR technology in the near future, where AR glasses will ultimately replace the smartphone as the dominant form of interface. With that in mind it is not hard to imagine how AR may be incorporated into dental education, including the traditional formats of lectures and didactic reading (Fig. 4). Advances in content creation engines such as the Unity game engine and Blender have made software development easier than ever before, giving dental educators an opportunity to develop new virtual and augmented reality dental education delivery and assessment methodologies that could enhance dental education.

Fig. 4

Oral Health welcomes this original article.

Correspondence: Dr. Akimasa Tsujimoto, Department of Operative Dentistry, Nihon University of Iowa College of Dentistry, 801 Newton Road, Iowa City 52242, USA. Tel: +1-319-467-4240; e-mail: akimasa-tsujimoto@uiowa.edu

References

1. Goodacre CJ. Digital learning resources for prosthodontic education: The perspectives of a long-term dental educator regarding 4 Key factors. J Prosthodont. 2018; 27(9): 791-797.
2. Mahrous A, Schneider GB, Holloway JA, Dawson DV. Enhancing student learning in removable partial denture design by using virtual three-dimensional models versus traditional two-dimensional drawings: A comparative study. J Prosthodont. 2019; 28(8), 927-933.
3. Riva G, Wiederhold BK, Gaggioli A. Being different. The transformative potential of virtual reality. Annual Review of Cybertherapy and Telemedicine. 2016; 14: 1-4.
4. Das P, Zhu Mo, McLaughlin L, Bilgrami Z, Milanaik RL. Augmented reality video games: new possibilities and implications for children and adolescents. Multimodal Technologies and Interaction. 2017; 1(2) 8.
5. Uruthiralingam U, Rea PM. Augmented and virtual reality in anatomical education – A systematic review. Adv Exp Med Biol. 2020; 1235: 89-101.
6. Kyaw BM, Saxena N, Posadzki P, Vseteckova J, Nikolaou CK, George PP, et al. Virtual reality for health professions education: Systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res. 2019; 21(1): e12959.
7. Farronato M, Maspero C, Lanteri V, Farma A, Ferrati F, Pottenuzzo A, et al. Current state of the art in the use of augmented reality in dentistry: A systematic review of the literature. BMC Oral Health. 2019; 19(1): 135.
8. Espejo-Trung LC, Elian SN, Luz MA. Development and application of a new learning object for teaching operative dentistry using augmented reality. J Dent Educ. 2015; 79(11): 1356-1362.
9. Liebermann A, Erdelt K. Virtual education: Dental morphologies in a virtual teaching environment. J Dent Educ. 2020; 84(10): 1143-1150.
10. Kim-Berman H, Karl E, Sherbel J, Sytek L, Ramaswamy V. Validity and user experience in an augmented reality virtual tooth identification test. J Dent Educ 2019; 83(11): 1345-1352. Mahrous A, Schneider GB, Holloway JA, Dawson DV. Enhancing student learning in removable partial denture design by using virtual three-dimensional models versus traditional two-dimensional drawings: A comparative study. J Prosthodont. 2019; 28(8), 927-933.

---

About the Author

Ahmed Mahrous is an Associate Professor and the Director of Digital and CAD/CAM Dentistry at A.T. Still University’s Arizona School of Dentistry & Oral Health. He qualified at Prosthodontics at the University of Iowa. He is the Creative Director of the Dental Atlas.

 

 

Amira Elgreatly is an Assistant Professor at A.T. Still University’s Arizona School of Dentistry & Oral Health. She qualified in Operative Dentistry at the University of Iowa. She is the Co-founder of the Dental Atlas.

 

 

Amr A. Mahrous is a Professor, Substitutive Dental Sciences Department, at Imam Abdulrahman Bin Faisal University College of Dentistry, Dammam, Saudi Arabia.. He qualified in dentistry at Alexandria University, and received PhDs in Conservative Dentistry, Endodontics, and Fixed Prosthodontics from the same university.

 

Akimasa Tsujimoto is an Associate Professor in the University of Iowa College of Dentistry. He qualified in dentistry at Nihon University, received PhD in Operative Dentistry and hold a tenured Associate Professorship in the same university and Honorary Professorship at Hong Kong University.