Abstract
Background. Malignant melanoma is among the fastest increasing malignancies in many countries. With the help of new tools, such as teledermoscopy referrals between primary healthcare and dermatology clinics, the diagnosis of these patients could be made more efficient. The introduction of a high-quality smartphone with a built-in digital camera may make the early detection more convenient. This study presents novel directions for early detection of malignant melanoma based on a smartphone application. Objectives and Methods. In this study, we concentrate on a precise description of a complex infrastructure of a fully automated computer-aided diagnostic system for early detection of malignant melanoma. The framework has been customized for a dermoscope that is customized to attach to the smartphone to be able to carry out mobile teledermoscopy. The application requirements, architecture, and computational methods as well as behavioral and dynamic aspects have been presented in this paper. Conclusion. This paper presents a broad application architecture, which can be easily customized for rapid deployment of a sophisticated health application. Mobile teledermoscopy is a new horizon that might become in the future the basis of the early detection of pigmented skin lesions as a screening tool for primary care doctors and inexperienced dermatologists.
1. Introduction
Human cutaneous melanoma, a malignant pigmented lesion, is the deadliest type of skin cancer. It is characterized by a rapidly rising incidence rate among Caucasian populations and every year tens of thousands of people worldwide die of this cancer [1]. Melanoma is among the most aggressive neoplasms and rapidly metastasizes to distant organs. When it progresses to metastatic stage, it establishes powerful mechanisms to resist chemo- and radiotherapy, thus hindering the efficacy of current medical therapies [2]. However, when detected early, melanoma is treatable in nearly all cases with a simple surgical excision [3].
Apart from that, there are also benign types of pigmented skin lesions, the so-called moles, that are natural parts of the skin. Both benign and malignant pigmented skin lesions share similar visual characteristics which makes differentiating between them a challenging problem for nonspecialists [4]. This issue is particularly significant during naked eye examinations, when early stage melanomas (Figure 1(b)) often resembles benign lesions (Figure 1(a)). Due to low public awareness of the importance of skin cancer prevention and insufficient access to dermatologists in many regions of the world, melanoma is often diagnosed only after a tumor grows to a medium size (Figure 1(c)).
(a) Clark nevus
(b) Melanoma in situ
(c) Melanoma 0.76–1.5 mm
In the light of the above data, prevention and early diagnosis of melanoma become extremely important issues. There is a demand to develop computer-aided diagnostic systems facilitating the early detection of melanoma which could be applied by nonexperts and the general public [5]. Based on our previous research, we propose a teledermoscopy system architecture to assess malignancy of a skin lesion as well as to differentiate between micromelanomas and developed skin moles.
The goal of this research is to provide future plans and directions for the early detection of malignant melanoma based on a smartphone application. We also summarize the state of the art in the teledermoscopy applications and outline our conclusions from previous researches. The article opens with a short introduction to the topic of the undertaken research, a clinical definition of malignant melanoma, and its epidemiology. It continues with a description of the new technology, mobile teledermoscopy (MTD), which has potential for early skin cancer detection and mortality rate reduction. Furthermore, a new robust review of applications for the detection and analysis of melanoma has been described. In Section 2, we describe a new approach to a fully automated computer-aided diagnostic system for early detection of malignant melanoma. The application requirements and architecture as well as behavioral and dynamic aspects are presented. In Section 3, the eSkin application is presented and the early experiments and results are described. In Section 4, the challenges involved in the processing of skin-lesion images acquired with mobile devices and implementation of a patient-oriented system for analyzing such images are outlined.
1.1. Clinical Definition
Melanoma (also malignant melanoma) is a malignant neoplasm, derived from cells that are capable of forming melanin, arising most commonly in the skin of any part of the body including eyes or even sore throat. The high concentration of the body’s pigmenting agent, melanin, is responsible for their dark color. Although melanocytic nevi are very common, their histogenesis is not well understood and still a matter of debate [3]. All we know about the life of melanocytic nevi is based on cross-sectional or cohort studies, because it is still complicated to monitor skin lesions in vivo on a cellular level. Skin moles may be congenital or developed during lifetime. The majority of moles appear during the first two decades of a person’s life. The congenital melanocytic nevus is more likely to develop into a melanoma, because of its larger size and influence of UV radiation or chemical substances. Smaller melanomas tend to develop sporadically from a pigmented nevus and occur most commonly in fair-skinned people. Any black or brown spot having an irregular border, pigment appearing to radiate beyond that border, blue, red, or white coloration observable on close examination, or a nodular surface is suggestive of melanoma and is usually excised for biopsy [9]. Melanomas are most commonly located on the upper back and lower legs of fair-skinned and on the palms of the hands and insoles of the feet of dark-skinned individuals. Melanomas may metastasize and are among the most malignant of all cancers. Figure 2 shows the stages of melanoma evolution process.
Prognosis depends on the kind of melanoma, its size, and location and depth of invasion. The most important parameter which predicts the stage of melanoma is the thickness of the examined lesion. Skin moles with the thickness less than 1 mm are nearly 100% curable [3]. Seidenari and coworkers reported the direct correlation between diameter and thickness, and as the diameter of in situ melanomas is smaller than that of invasive lesions, it is reasonable to believe that small melanomas are usually in an initial growth phase [10]. Therefore, the aim of each clinician is to detect malignant melanomas when they are still small and thin. Figure 3 shows dermoscopic images of successive stages of melanoma.
1.2. Epidemiology of Melanoma
Over the past several decades, there has been a significant increase in the incidence and mortality rate from skin cutaneous melanoma among Caucasian populations worldwide (Figure 4) [1]. Despite the fact that only about 4% of all diagnosed skin cancers are melanoma, melanoma is responsible for about 70% of skin cancer-related deaths in the United States and in Australia [11, 12]. Invasive melanoma has an estimated incidence of 73,870 and an estimated total of 9940 deaths in the United States in 2015 [13].
One of the most important factors considered to result in melanoma is the brief, intense sun exposure pattern. Due to a constant depletion of ozone layer in stratosphere, which results in higher exposure to UV radiation, malignant melanoma is likely to become one of the most common malignant tumors in the future, with even 2–10 times higher incidence rate [1, 3]. As no effective treatment of melanoma in advanced stages has been developed so far, its early diagnosis has become an extremely important issue.
1.3. Improving Melanoma Detection Based on Sensing Technologies
Telemedicine which is also called telehealth, online health, e-health, or “medicine at a distance” is a new and rapidly developing field of medicine, providing access to medical knowledge that would be not available at a particular location and time. It is commonly defined as the use of telecommunication technologies for the exchange of medical information over a distance for the purpose of patient management (including triage, diagnosis, and therapeutic suggestions, as well as follow-up) and medical education [14]. Teledermatology has been listed in Norway among priority telemedicine specialities for large-scale implementation underlining the growing interest in this field [14, 15].
The most important question that has to be posed while supporting the early skin cancer detection is does the teledermoscopy system achieve an adequate accuracy and is it effective? In this research, we will analyze only two factors including diagnostic reliability and accuracy as well as economic analysis. With no doubt, the most significant parameter while using any computer-aided diagnosis system is the diagnostic reliability and accuracy which have to be compared between the diagnostic system and conventional method (human interpretation) [16]. Diagnostic accuracy assessments for dermatologic disease are still problematic. The histopathological examination constitutes the gold standard for classifying a melanocytic skin lesion as malignant. Although there exist other forms of examination (e.g., dermatoscopy), they yield lower diagnostic confidence than a histopathological examination. Still, numerous studies showed that the misdiagnosis rate of melanoma may be as high as 10–25%. Histopathologic review cannot be universally used to make a definitive diagnosis of all skin lesions. Despite this, few studies have analyzed data among patients that underwent biopsy and other diagnostic methods which made it possible to compare the teledermatology systems with the clinic-based examination [16–18]. Teledermatology systems achieved accuracy at about 70–85% which indicates that the computer-based solutions have comparable diagnostic accuracy when compared to conventional clinic-based care [16]. While implementing and deploying a teledermatology system, the economic aspects play an important role. The following criteria are taken into account: dermatology visits that have been avoided, skin biopsies skipped, and consult costs. In study [19], authors have evaluated the proportion of real-time interactive teledermatology consults that avoided a clinic-based visit to a dermatologist. Dermatology clinic visits have been avoided in 54% thanks to the real-time interactive teledermatology. Also the decrease of the number of biopsies is a cost-saving strategy. Based on a quick evaluation of the dermoscopy image, inexperienced physicians will gain knowledge and avoid unnecessary treatment.
1.4. Related Works
According to the estimates by the American Food and Drug Administration, nearly 500 million smartphone users worldwide use eHealth mobile applications. In July 2014, there were 39 dermatology-related mobile applications aimed at general community, patient, and generalist clinician users available on the market [20]. The main functionality of over half of them was to provide information or education about melanoma, UV radiation exposure prevention advice, and skin self-examination strategies. Such applications are usually targeted at students and novice doctors. Another large group of applications were those capable of taking images of moles using mobile embedded camera, tagging them with the body location, and storing them either for review by a dermatologist or for self-monitoring to track changes in mole appearance, an important predictor of melanoma.
In March 2017, there were over 45 mobile applications related to mole diagnosis available on Apple’s App Store alone. Most of them () offered only educational information on melanoma, nearly half of them () allowed the user to take photos of their moles and track changes over time using simple visual comparison, and only four applications performed melanoma risk assessment or lesion classification based on image analysis (Table 1).
Only the authors of SkinVision and Lūbax made public the results of clinical evaluation of their risk assessment algorithm [21, 22]. Out of 4 applications summarized in Table 1, only two were certified by authorities: SkinVision received the European “CE” Marking and DermaCompare was approved by the US Food and Drug Administration.
1.4.1. SkinVision
The risk assessment algorithm used by SkinVision is based on the analysis of a gray-scale image of a lesion () and its associated fractal map . The fractal map is generated based on the weighted local fractal dimension (WLFD) proposed in [23] (the WLFD was originally used for computer tomography image enhancement, but authors of SkinVision adapted it to the context of dermatoscopy). To segment the lesion (and thus extract the lesion contour), thresholds and were obtained by applying Otsu’s thresholding on and , respectively. The following parameters are calculated on : percentage of pixels with WLFD lower than , percentage of pixels with WLFD lower than , number of connected regions with distinct textures found in the interval , and , and the following are calculated on : number of connected regions with distinct intensities found in the interval , , and circularity index. Based on these parameters, the SkinVision application evaluates lesions to one of the three risk classes: high, medium, or low.
SkinVision was tested on a set of melanocytic lesions images taken using iPhone 4S mobile device equipped with an 8-megapixel autofocus camera. It achieved the overall sensitivity of 73%, specificity of 83%, and accuracy of 81%. The positive and negative predictive values were 49% and 83%, respectively.
1.4.2. Lūbax
The skin-lesion classification system used by Lūbax consists of a proprietary database of nearly 12,000 images of diagnosed lesions and a computer algorithm based on the principles of content-based image retrieval (the algorithm compares characteristics of new images with images in the database to identify the nearest-match diagnosis).
The lesion images were taken using Celestron® (Torrance, CA) hand-held digital microscopes equipped with 2-megapixel cameras with a macro lens surrounded by a ring of white light-emitting diode lights. The consistent lighting conditions and imaging distance were ensured by attaching an opaque 10 cm tube to the front of each camera. The largest diameter for all nonmelanoma and melanoma lesions was at least 10 mm.
Each lesion image in the database was reviewed and diagnosed by at least one of three board-certified dermatologists using standard clinical criteria. The “ground truth” classification was not biopsy-proven, but based on the agreement between the reviewing dermatologists. Only 302 images of melanoma borrowed from the DermNet NZ [24] database were previously confirmed by histopathology.
The algorithm compares new images of skin lesions with the database of diagnosed skin-lesion images. It uses orientation- and artifact-independent image information on lesion size, color, shape, and texture to create a single high-dimensional signature for each image. The “malignant versus nonmalignant” classification is then performed using a -nearest-neighbor classifier.
Lūbax was evaluated on a set of 337 images queried from the database, out of which 208 were melanomas. It scored sensitivity of 90.4%, specificity of 91.5%, and accuracy of 90.8%. The positive and negative predictive values were 94.5% and 85.5%, respectively.
2. Development of the eSkin Teledermatoscopy System
Based on the above information, there is still a great need in the development and analysis of teledermatoscopy systems. In this paper, we propose a new approach to the analysis of nonmelanoma and melanoma lesions. The innovation is firstly based on the differentiation between micromelanomas which have diameter less than 5 mm and advanced skin moles. Secondly, we propose an additional algorithm for the skin-lesion differentiation that can be applied in a teledermoscopy system and has been described in our work [25]. In this section, we present the system overview and describe the technical requirements, application architecture, and behavioral and dynamic aspects of the system.
2.1. Concept of the System
The purpose of presenting the concept of the system is its better illustration and description of the relationships between the individual modules of the system. While composing the system, overview issues such as safe medical information transfer, fast connection availability, server choice, and database type are analyzed. Figure 5 presents the implemented concept of a system that consists of a smartphone application available to the client (client side) and the server side of the system, consisting of a general-purpose server, computing server, users database, and images database.
2.2. Application Requirement Specification
The most important piece of hardware is a high-quality camera system. Certain dermoscopic features of a lesion, such as pigment network or dots, are clearly visible only under magnification (usually of at least ×). Moreover, since colors visible in a lesion are important diagnostic clues, it is desirable to take images in appropriate illumination conditions.
Most modern smartphone cameras (this is true, e.g., for all iPhone devices starting from iPhone 5S and all Samsung Galaxy devices starting from Samsung Galaxy S4) are equipped with at least 8-megapixel sensor whose pixel size is at most 1.5 m, allowing the user to capture high-quality photos. However, smartphone cameras generally lack a quality optical zoom and are not equipped in a source of white light which would uniformly illuminate an examined lesion.
The solution to the above-mentioned deficiencies of bare smartphones is to use either a smartphone dermatoscope or a conventional dermatoscope with a mobile phone case (Figures 6 and 7). A smartphone dermatoscope is an attachment for dermoscopy that provides a detailed view of the skin through magnification and specialized lighting. Most smartphone dermatoscopes currently available on the market provide × to × magnification, work in both polarized and nonpolarized lighting modes, and are capable of performing examinations in both contact and noncontact mode. Some conventional dermatoscopes may be attached to a mobile phone using a special case (e.g., all dermatoscopes from 3Gen’s DermLite and Canfield’s VEOS series may be used with iPhone 5 and newer and 3Gen’s DermLite and MoleScope products can be used with Samsung Galaxy S4 and newer).
(a) MoleScope™
(b) DermLite® Connection Kit
(a) No attachment
(b) HÜD™
(c) MoleScope II™
2.3. Application Architecture
Like any other complex system, software and essentially medical applications have to be built on solid foundations. During the smartphone application design process, it is important to define a structured solution that meets all of the technical and operational requirements, while optimizing common quality attributes such as performance, security, and manageability. The process involves a series of decisions based on a wide range of factors, and each of these decisions can have considerable impact on the quality, performance, maintainability, and overall success of the application [26]. The proposed system consists of two parts: mobile application and servers (Figure 8).
The tasks of the mobile application include the following:(i)Friendly user interface with the ability to log in and change basic settings(ii)The ability to take a picture of a skin mole or select from already saved pictures on the smartphone(iii)Presentation of results of classification of selected skin mole(iv)User location for UV hazard rating(v)Communication with servers including general-purpose server, image database, user database, and computing server(vi)Checking the Internet connection and data security.
The tasks performed by the server include the following:(i)Communication with the user mobile device (general server task)(ii)User login (username and password verification) and access control (general server and user database task)(iii)Calculation of feature parameters, image classification, and comparison of ratings (computing server task)(iv)Medical data archiving (medical images, computed skin mole parameters, and classification results) (image and user database task).
2.4. Behavioral and Dynamic Aspects of the System
The activity diagram of the proposed mobile application, capturing its dynamic behavior, is shown in Figure 9. The main functionality of the application is to take photos of moles and send them to the server in order to perform melanoma risk assessment. It is also capable of archiving captured photos and browsing through past examinations. Additional features, intended to rise melanoma awareness, include ultraviolet radiation risk alerts, reminders about a subsequent examination, and a tool to easily find the nearest dermatologist.
A data flow diagram (DFD) is a graphical representation of the “flow” of how data is processed by a system in terms of inputs and outputs. It focuses on the flow of information, where data comes from, where it goes, and how it gets stored. It is a picture of the movement of data between external entities and the processes and data stores within a system. Figure 10 presents the proposed DFD diagram. A DFD diagram consists of external entities, processes, and data stores:(i)External entities: origin or destination of data (outside the system): user (mobile application), computing server, and general-purpose server(ii)Processes: work or action performed on data: image evaluation, authentication, image storage, and subcontract image evaluation(iii)Data stores: data that the system stores: users’ database and images’ database.
2.5. Computational Methods
Analysis and classification of melanoma skin lesions are a complex issue due to the different appearance on various levels of disease progression. Our recent research and experience in the topic of early detection of melanoma have shown that different approach to the classification process should be undertaken to improve the recognition results. The implemented and tested computer-aided diagnostic system has been presented in Figure 11. The first two steps including image preprocessing and segmentation are performed for each of the analyzed medical pictures.
The preprocessing step is essential for dermoscopic images to improve the quality due to the extraneous artifacts, such as skin lines, air bubbles, and hairs which appear in almost every image. The preprocessing step contains three algorithms [27, 28]:(i)Black frame removal: black frames, introduced to the image during its digitization, are detected using lightness component of the HSL color space.(ii)Smoothing: Gaussian filter is used for smoothing of air bubbles and light hairs.(iii)Black hair inpainting: for removing black and thick hairs, we chose the white top-hat transform. Hair line pixels are replaced with values calculated on the basis of the neighborhood pixels.
A medical image is one of most complicated images to be segmented; furthermore, this step is crucial for sequential analysis and diagnosis. During our work, we have compared different segmentation methods and on the grounds of the previous outcomes the applied segmentation algorithm for the skin-lesion extraction is based on seeded region-growing algorithm [29, 30].
After the first two steps, the segmented skin mole is ordered to one of the two classification algorithms. The division is done on the basis of the mole diameter. Skin moles with a diameter lower than 5 mm are passed to the Micro-Melanoma algorithm and the remaining ones to the Specific Melanocytic Lesion algorithm. The diameter of the lesion is provided by the user as it is not possible to assess it only based on the medical image. Micromelanomas represent a minority of diagnosed lesions, their frequency ranging from 1% to 17%. The mean diameter of in situ melanomas is around 1 cm, and invasive melanomas are usually greater than 6 mm. Despite the fact that micromelanomas are in the minority, they are responsible for most errors during the diagnosis. Separation of individual changes allows us to obtain a more accurate and reliable computer diagnosis system. The Micro-Melanoma algorithm and the Specific Melanocytic Lesion algorithm have been described in detail in [25, 28].
3. Early Experiments
We implemented the application on an iOS smartphone, which most of the people in US use in their daily lives. This small smartphone includes all sensors that can provide enough information needed for realizing this system. The preliminary version of the eSkin application has been implemented in Swift in the Xcode development environment which is dedicated for macOS and contains a suite of software development tools for macOS, iOS, watchOS, and tvOS. Figure 12 presents the proposed version of the graphical user interface. The basic functionality including taking a medical photo and assessing the skin mole as well as medical data archiving have been implemented.
The eSkin application uses a scientifically proven algorithm to detect and analyze the dermoscopy images for visible signs of skin cancer. To measure the diagnostic performance, we calculated sensitivity, specificity, and the area under that plot curve (AUC). The fundamental definitions of these performance measures could be illustrated as follows.
The sensitivity (also called True Positive rate) is the probability that the cell is said to be adherent given that it is. This can be estimated by relative frequency: where is the True Positive answer and is the False Negative answer. The value of sensitivity ranges between 0 and 1, where 0 and 1, respectively, mean worst and best classification.
Specificity (also called False Positive rate), which is defined as the proportion of the True Negatives against all negative results, is defined by the following equation: where is the True Negative answer and is the False Negative answer. Value of specificity ranges between 0 and 1, where 0 and 1, respectively, mean worst and best classification.
Table 2 shows the classification results obtained with the implemented algorithms for microlesions as well as developed skin moles.
For statistical analysis, SkinVision classified histologically proven nevi as low or medium risk and melanoma as high-risk lesion. The statistics included 144 lesions (with a minimum of 3 images per lesion) with the following histological diagnosis: 84 benign nevi (58%), 34 dysplastic nevi (24%), and 26 melanomas (18%).
Images of poor quality (e.g., due to inappropriate imaging angle or distance) and containing other elements not belonging to the lesion (e.g., hair), as well as cases with an equal number of results in two consecutive risk classes (e.g., 1 high risk, 1 medium risk, and 1 low risk result), were excluded from the analysis. In total, 26% of the images initially taken have been dismissed due to improper imaging.
The sensitivity of the SkinVision melanoma detection algorithm compared to the histological result was 73%, the specificity 83%, and the accuracy 81%.
The performance of the melanoma detection algorithm by Lūbax was assessed on a set of 337 images randomly selected from their database. The set included 129 images of nonmelanoma lesions and 208 images of melanoma lesions, all with the largest diameter of at least 10 mm. All melanoma query images were selected from the set of images acquired from DermNet NZ to ensure confirmation of the malignancy by histopathology.
The sensitivity of the Lūbax melanoma detection algorithm was 90%, the specificity 91%, and the accuracy 91%. For the melanoma classification, the sensitivity of the SkinVision algorithm compared to the histological result was 73%, the specificity 83%, and the accuracy 81%.
The conducted results achieved by the implemented classification system are much better than the results described in similar works. Our method allowed classification of skin moles very precisely and we can confirm that the classification of micro-skin lesions has to be done separately from the classification of developed skin moles. Overall, our experiments clearly show that the classification can be supported by a powerful system based on advanced machine learning techniques.
4. Conclusions and Discussion
In this paper, we propose a new teledermatology system for melanoma diagnosis, based on our previous research on assessing malignancy of a skin lesion as well as on differentiating between micromelanomas and developed skin moles. Our system is built in a client-server architecture and is intended for smartphone users.
In our system, the main functionality of the client application is to take photos of moles and send them to the server in order to perform melanoma risk assessment. By shifting all computations related to image assessment to the dedicated servers, we may analyze dermoscopic images using computationally demanding algorithms. Since the medical data would be archived in a central database, they would provide invaluable aid when working on improvements to the diagnostic algorithm. The client-server architecture would also make the process of implementing changes to the assessment algorithms seamless for the users, since there would be no need for client software updates.
To ensure high quality of images and good visibility of certain dermoscopic features of a lesion, the system is designed to work with smartphone dermatoscopes or conventional dermatoscopes with a mobile phone case. Such a solution would also reduce the negative impact of taking images in appropriate illumination conditions and make it possible to avoid issues with determining the size of a lesion. The latter issue is of particular importance, since our recent research has shown that depending on the size of a lesion different diagnostic algorithms should be used to improve the recognition results.
Although the client side of our system was initially developed for iOS devices, it does not use any functionality specific to either iOS or Apple’s smartphones. Therefore, it could easily be ported to work with devices running the Android operating system.
Due to low public awareness of the importance of skin cancer prevention and insufficient access to dermatologists in many regions of the world, there is a demand to develop computer-aided diagnostic systems facilitating the early detection of melanoma which could be applied by nonexperts. However, as of March 2017, there were only four mobile applications performing melanoma risk assessment or lesion classification based on image analysis available to the general public. With teledermatology systems achieving accuracy up to 85%, comparable to conventional clinic-based care, we are convinced that our system would fulfill that demand and allow its users to avoid dermatology visits, to skip skin biopsies, and to cut costs of consultations.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This scientific work was supported by AGH University of Science and Technology research grant on Decision nos. 15.11.120.883 and 15.11.120.635 to J. Jaworek-Korjakowska.