1. Recent trends in AI research

It is well-established that automated analytical systems have begun to emerge as a database system capable of scanning medical images with computers and configuring big data. With the development of back-propagation deep-learning artificial intelligence (AI) architectures, medical imaging has begun to be applied and it has been reported that accurate diagnosis is possible. The most efficient model for image analysis is the ‘convolutional neural network’ (CNN), a key block in the configuration of deep networks. CNN involves developing various optimization algorithms such as LeNet, AlexNet, ZFNet, GoogLeNet, VGGNet, and ResNet. CNN is a powerful feature extractor with a deep layer that can extract features from images.^1,2) Deep learning algorithms, especially convolutional networks, are rapidly emerging as a methodology for analyzing medical images. There has been remarkable growth in medical imaging analysis of the nerves, retina, lung, digital pathology, breast, heart, abdomen, and musculo-skeletal system using the AI methods that make up the deep-learning architecture.

2. The need for AI in medical imaging

Finding non-invasive and quantitative assessment techniques for early detection of Alzheimer’s disease (AD) is fundamentally important for early treatment. Tumor detection, classification, and quantitative assessment in positron emission tomography (PET) imaging are important for early diagnosis and treatment planning. A number of techniques have been proposed for segmenting medical image data through quantitative assessment. However, some quantitative methods of evaluating medical images are inaccurate and require considerable computation time to analyze large amounts of data. Analytical methods using AI algorithms can improve diagnostic accuracy and save time.

3. AI research field

The deep-learning technology, known as the AI method, dramatically improves diagnostic performance by automatically extracting features of complex and precise medical images, and comparing their differences.^3,4) Image analysis by AI algorithm is superior to the traditional image analysis method. Automatically classifying skin lesions using images is a challenging task because the skin lesion shapes differ a lot, but they have successfully categorized skin cancer successfully.⁵⁾ When evaluating diabetic retinal fundus photographs, deep-learning using Inception-v3 architecture enabled diagnosis of diabetic retinopathy with high sensitivity and specificity.⁶⁾ Early detection of pulmonary nodules on a chest by computed tomography (CT) scan was performed using the lung image database consortium (LIDC) and the image database resource initiative (IDRI) for the diagnosis of lung cancer, and extracted successfully using the network back propagation algorithm.⁷⁾ CNN-type deep-learning model, trained for large mammography lesions, is superior in performance to existing computer-aided diagnosis (CAD) system.⁸⁾ An automated detection system has been successfully established by studying the feasibility of a deep-learning approach for detecting cartilage lesions in the knee joint with magnetic resonance (MR) images.⁹⁾ PET images with low spatial resolution overestimated the volume due to the partial volume effect, but the optimal volume was extracted using the artificial neural networks algorithm.¹⁰⁾ Through the development of AI algorithms, the diagnostic performance of various medical images has been improved, and it is expected to be continuously introduced into medical image diagnosis systems because it has a higher diagnostic performance index than any other quantitative analysis methods so far.

For the diagnosis of medical imaging, there should be labeled and standardized big data, as shown in Fig. 1. AI is generally a concept that includes machine-learning and deep-learning, and can be analyzed in various ways, depending on the characteristics of images stored in the database. The data is pre-processed in the same order as in Fig. 2, entered into the deep-learning architecture, and final diagnosis results are obtained. The hyper-parameter is changed and the deep-learning result is checked and modified to optimize the parameter. In order to solve the problem of over-fitting due to lack of the sufficient labeled data, data augmentation is performed through rotation, left and right flip, and generative adversarial network (GAN).

1. Data sets

Although medical images are stored in the picture archiving and communication system (PACS), they are almost useless data for AI analysis. In order to use medical images for AI, labeling, standardization, bounding box, and segmentation are required. These high quality images require manual work, which can be time-consuming and can vary, depending on the skill of the person. Standardized and anonymized well-labeled databases are very important for developing and testing AI algorithms that require big data. There has been an attempt to build a large-scale database system through these complex processes, and to improve diagnostic performance through competition. These attempts have been quite successful and appear to be most effective in increasing diagnostic accuracy.

In recent years, an anonymous medical image database has been released and researchers are freely approaching and presenting research results. A database of F-18 fluorodeoxyglucose (FDG) PET images from participants of the AD neuro-imaging initiative (ADNI) is being developed to study early detection of AD patients. F-18 FDG PET images of the ADNI database were used to distinguish AD patients with 88.64% specificity, 87.70% sensitivity, and 88.24% accuracy.¹¹⁾ Magnetic resonance imaging (MRI) of the knee is the preferred method for diagnosing a knee injury, but the analysis is time consuming and the likelihood of diagnostic errors is high. MRNet, a CNN, and logistic regression analysis were performed as a dataset of 1,370 knee MRI scans performed at the Stanford University Medical Center (https://stanfordmlgroup.github.io). The results of the validation did not show a significant difference from the analysis of 9 clinical experts at the Stanford University Medical Center.¹²⁾ The leader-board reports the average area under the receiver operating characteristic curve (AUC; 0.917) of current (Jan 09, 2019) abnormal detection, anterior cruciate ligament tears, and meniscal tear work (https://stanfordmlgroup.github.io/competitions/mrnet).

To screen for severe pneumothorax, a total of 13,292 frontal chest X-rays were visually labeled by a radiologist to create a database.¹³⁾ Candidate images for analysis were identified and stored in a clinical PACS in the form of ‘digital imaging and communications in medicine’ (DICOM; National Electrical Manufacturers Association, Arlington, VA, USA). These well-defined data sets can be used to train and evaluate different network architectures. Building such a well-defined database requires a lot of time and effort. The National Institute of Health (NIH) has opened the Chest X-ray14 set, so that it is publicly available (available at https://nihcc.app.box.com/v/ChestXray-NIHCC). The NIH allows students to learn by using well-organized and labeled data sets, and opens up a database for anyone to challenge and develop optimized algorithms. LIDC and IDRI databases were created and released for the diagnosis of lung cancer. For the diagnosis of skin cancer, the database can be opened (https://isic-archive.com) to participate in the competition.⁵⁾ The MR knee database is also available and anyone interested in this field can participate in AI analysis (https://stanfordmlgroup.github.io/competitions/mrnet). Most problems with large data sets and public challenge data sets are listed at http://www.grand-challenge.org.

In order to construct a more efficient accurate, and useful database, the Department of Nuclear Medicine at Dong-A University Medical Center established a database system called SortDB with IRM (http://www.irm.kr/). The feature of SortDB is some software that allows researchers to download a desired DICOM file in a specified file format (nii, jpg, gif, etc.) by entering a manageable list of Excel files. To apply an AI algorithm, big data is needed. It is an automated database generation program that stores and manages data stored in small and medium sized hospitals in the necessary format. A database is constructed with the system configuration shown in Fig. 3, shortening the pre-processing time for creating a standardized data set.

2. Pre-processing of medical images

For quantitative analysis, an image was normalized using a standard affine model with 12 parameters using SPM5 software.¹⁴⁾ In order to use quantitative evaluation using machine learning, a pre-processing method, including the normalization process of image data must necessarily be included. Since the accuracy of the normalized image depends on the standard template or the variety of the process, there are considerable variations among researchers. When applying an AI algorithm through a normalized image, the accuracy is generally increased. However, there is a lack of logical explanations and studies on why normalized images increase accuracy. Theoretical studies are needed, as well as empirical studies on the optimal pre-processing method for image analysis using AI.

The accuracy of the quantitative analysis has a considerable impact on the quality of the image transformation by pre-processing. In particular, the accuracy of AI analysis varies considerably, depending on the quality of medical images for applying the algorithm. The image transformation of the DICOM file is linearly transformed using the return value of GetOuputData function of a library.¹⁵⁾ Bits number 8 and the standardized uptake values (SUV) were linearly converted to 0–255 scales. By using this method of linear transformation, the image representing the SUV can be reproduced as a jpg file with distortion. It is necessary to study the image conversion methods so that medical images will suffer only minimal deformation in the conversion process. It is necessary to maintain the consistency of AI learning using image-aware CNNs by down-sampling commonly used images.¹³⁾

3. Data augmentation, model and parameter optimization

Generally, PET images do not have a large number of data, making it difficult to construct a database of big data. In this case, data augmentation is performed through image inversion, enlargement/reduction, shearing, and rotation to randomly distribute data to perform AI analysis. Too many hyper-parameter optimizations are some of the most important difficulties in achieving optimal accuracy. This can be done using the open source Future Gadget Laboratory (https://github.com/Kaixhin/FGLab) framework.¹³⁾ Evaluation of the validity of deep-learning can be done by calculation of the AUC, sensitivity, specificity, and positive predictive value of the entire validation set, and can be done using scikit-learn (https://scikit-learn.org). Receiver operating characteristic curves can be drawn using matplotlib (https://matplotlib.org).

Currently, CNN algorithms are implemented in various ways. The recent trend is to use Keras (version 2.0.3, https://keras.io/) deep-learning library on TensorFlow (version 1.2.1, https://www.tensorflow.org/?hl=ko) to implement a model of CNN. Deep-learning architectures for AI analysis are VGG16/19, Xception, Inception, and ResNet models, which are the best performing algorithms.^16-19) The optimal algorithm for image classification may be an algorithm that can reflect the feature characteristics of each medical image. Currently, the location of such features is estimated through the activation map. However, at present, only the experience of repeated execution of each medical image is the method of finding the optimal model. The AlexNet algorithm, which won the ImageNet Large Scale Visual Recognition Challenge in 2012²⁰⁾ to classify the chest X-ray image into 5 classes, was used and successfully classified with 92.10% accuracy.²¹⁾ AlexNet consists of 5 convolution layers and 3 fully-connected (FC) layers, and the last FC layer uses a softmax activation function. The researcher²¹⁾ down-sampled the DICOM file and converted it into a 256×256 pixel png file and used it as the input image of AlexNet. The mini-batch size is 128, the number of training iterations is 100, the adaptive learning rate is 1×10⁻³ for adaptive moment estimation (Adam) optimizer, and the momentum is 0.5. The rectified linear unit (ReLU) activation functions were used in the previous step of the max-pooling layers and the L₂ regularization was adjusted to 1×10⁻⁴.

One of the most successful methods is the principal component analysis (PCA) method, which is based on the eigen-image approach.²²⁾ After analyzing the images through PCA, support vector machine (SVM) was used as a classifier to distinguish the boundary between normal and demented patients and to improve the accuracy by diagnosing and analyzing the initial dementia. PCA was applied to single photon emission computed tomography (SPECT) images and the possibility of classification was greatly improved, and the results were presented through 3-dimensional (3D) scattering.²³⁾ Quantitative evaluation and classification is one of the most efficient and logical methods. The accuracy of classification of AD by applying PCA-SVM to SPECT and PET images was 96.7% and 89.52%, respectively.²⁴⁾ Using Gaussian mixture models, a method of automatically selecting the region of interest (ROI) of a 3D functional brain image containing high and low activation region information was presented.²⁵⁾ Based on classification using eigenvector decomposition and SVM, including concept extraction by representative vector separation using principal component analysis, we find that PCA/independent component analysis for early AD diagnosis by brain image analysis and pattern disease discrimination CAD is a very useful direction.¹¹⁾

1. AI in brain imaging

In brain research using AI, many studies have been conducted in the field of AD classification, anatomical segmentation of brain regions, and tumor detection. AD/mild cognitive impairment/healthy control classification was successfully performed by using the Gaussian restricted Boltzmann machine (RBM) to find feature expressions in volume patches of MRI and PET images.²⁶⁾ The 3D CNN in AD classification is superior to other algorithm classifiers.^27,28) It automatically segments the MR images of the human brain using CNN.²⁹⁾ Segmentation of striatum segmentation was performed using deep CNN, and the results were compared with those of FreeSurfer.³⁰⁾ In the brain area, manual segments are time-consuming, with individual differences occurring, while automatic segmentation has significant difficulties in performing in complex structures. Twenty-five deep-layers called the ‘voxel-wise residual network’ (VoxResNet) were developed and successfully segmented automatically.³¹⁾ To demonstrate end-to-end nonlinear mapping from MR images to CT images, a 3D fully CNN was employed and verified in a real pelvic CT/MRI data set.³²⁾ Input and output improved performance by using two-volume CNNs, and excellent performance was observed by evaluating the input and output forms in the MRI and PET images of the ADNI database.³³⁾

2. AI in chest imaging

By introducing the multiple-instance learning framework, a de-CNN is constructed to generate the heat map of suspicious regions.³⁴⁾ A unique set of radiologic datasets of publicly available chest X-rays and their reports were used to find and report 17 unique patterns by applying CNN algorithms.³⁵⁾ It has been reported that the presence of interstitial patterns have been found by applying a segmentation-based label propagation method to a dataset of interstitial lung disease,³⁶⁾ and it has been reported that lung texture patterns are classified using CNN.³⁷⁾ A method for classifying frontal and lateral chest X-ray images using deep-learning methods and automating metadata annotations has been reported.³⁸⁾ A new method of using a 3D CNN for false positive reduction in automatic pulmonary nodule detection in a volumetric CT scan has been proposed. 3D CNN is able to enter more spatial information and extract more representative features through a hierarchical architecture, trained with 3-D samples. The proposed algorithm has achieved high competition metric scores, has been extensively tested in the LUNA16 Challenge, and can be applied to 3D PET images.

3. AI in breast imaging

Since most mammograms are 2D and the number of data is large, AI images can be successfully analyzed using deep-learning in natural images. The discovery of breast cancer is the detection and classification of tumor lesions, the detection and classification of micro-calcifications, and risk-scoring work, which can be effectively analyzed by CNN or RBM methods. For the measurement of breast density, CNNs for feature extraction were used,³⁹⁾ and a modified region proposal CNN (R-CNN) has been used for localization. It has been reported that U-net is used for segmentation breast and fibro-glandular tissue in MRI in a dataset and accurate breast density calculation results are observed.⁴⁰⁾ It has been reported that a short-term risk assessment model has been developed by achieving a predictive accuracy of 71.4% by calculating the risk score by implementing the mammographic X-ray image as a risk prediction module multiple layer perception classifier.⁴¹⁾

4. AI in cardiac imaging

Cardiac artificial research fields include left ventricle segmentation, slice classification, image quality assessment, automated calcium scoring, coronary centerline tracking, and super-resolution (SR). 2D and 3D CNN techniques are mainly used for classification, and deep-learning techniques such as U-net segmentation algorithm are used for segmentation. The high-resolution 3D volume in the 2D image stack has been reconstructed using a novel image SR approach.⁴²⁾ The image quality is superior to the SR method because the CNN model is computationally efficient, but SR-CNN is advantageous in image segmentation and motion tracking.⁴³⁾ Using multi-stream CNN (3 views), it has been reported that low-dose chest CT can be identified with high accuracy by deep learning when the ROI is considered as a coronary artery calcification candidate over 130 HU.⁴⁴⁾ Coronary calcium in gated cardiac CT angiography was detected using 3D CNN and multi-stream 2D CNN.⁴⁵⁾

5. AI in musculo-skeletal imaging

Musculo-skeletal images are analyzed by deep-learning algorithms for segmentation and identification of bone, joint, and associated soft tissue abnormalities. 3D CNN architecture has been developed to automatically perform supervised segmentation of vertebral bodies from 3D MR spine images, and a ‘Dice’ similarity coefficient of 93.4% has been reached.²³⁾ Automatic spine recognition, including spine location identification and multiple image naming, requires large amounts of image data, and is difficult to recognize due to the variety of spine shapes and postures. By using a deep-learning architecture called transformed deep convolution network, the posture of the spine was automatically corrected to process the image.⁴⁶⁾ A CNN regression has a limitation in that its computation time is long and the capture range is small for implementing an intensity-based 2D/3D registration technology. It is reported that highly accurate real-time 2D/3D registration is possible, even in a greatly enlarged capture range.⁴⁷⁾ Several deep-learning methods have been developed for automatically evaluating the age of skeletal bones using X-ray imaging and their performance has been verified by showing an average discrepancy of about 0.8 years.⁴⁸⁾

Diagnosis using AI is quickly performed and its accuracy is very high. AI diagnosis is becoming an important technology for future diagnostic systems. However, AI diagnosis needs to be supplemented in several aspects. AI learning using deep-learning architecture requires big data. However, most medical images are technical and man-powered, making it difficult to build big data systems. It is also time-consuming to create a database of standardized and labeled images of medical images. Most people are building databases by manually pre-processing all medical images for AI application. Performing data augmentation through rotation, left/right flip, and up/down reversal of a medical image has a positive effect on the accuracy of learning. Data augmentation using GAN is being applied in various areas of the medical field.^21,49) In liver lesion classification using CT images, it was reported that the accuracy of 7.1% was increased when the number of data using GAN was increased.²¹⁾ In chest pathology classification using X-ray image, accuracy was reported to be increased by 21.23%.⁴⁹⁾ It has been reported that synthesized images using GAN can be a method of data augmentation in medical image analysis. However, more research is needed to see if synthetic images can be used for AI learning to determine clinical diagnostics that require rigorous accuracy. Increasing data through the GAN is still controversial, but it is a field of research that is needed. It is necessary to make the medical image stored in the PACS system into the image of the previous step for AI analysis. Research is needed to automatically generate standard images so that medical images can be used directly in deep-learning.

AI analysis of medical images requires labeled, standardized and optimized images. There is a clear difference in the accuracy of the final classification between pre-processing and non-pre-processing medical images. However, since the pre-processed image has noise different from that of the original image, it is necessary to study the effect of the resulting image on the accuracy. Deep-learning architecture can have various forms, and new excellent architecture for image analysis is being released every year through competition. However, it is only through experience that the most accurate result of any architecture can be used for each medical image. Therefore, it is necessary for people to apply various deep-learning architectures to each medical image and then to discuss each other’s experiences to find the optimal method. Even if one deep-learning model is determined, there are many hyper-parameters in it. The composition of parameter combinations is often dependent on user experience. Optimization of Hyper-parameters is usually done through a time-consuming grid search and random search, but Bayesian optimization and genetic algorithm optimization can be used efficiently.

Medical imaging diagnostics using deep-learning architecture have reached expert levels in the areas of neurons, retina, lung, digital pathology, breast, heart, abdomen, and musculo-skeletal system. However, when other hospital protocols apply different medical images, their accuracy is significantly reduced and new optimization parameters must be found. Even with some resolution and noise changes in medical imaging, AI diagnostics are very fragile. There is also a lack of logical explanations as to which process to diagnose. Although there is an effort to maintain consistency through data harmonization, there is the possibility that the overall quality of the image will deteriorate. It is necessary to study the change of the accuracy by using the data whose resolution is decreased. It is necessary to develop algorithms that can recognize and generalize images of medical images with different resolutions or noise, but it will take a considerable amount of time. Nevertheless, the diagnosis of medical images by the deep-learning architecture developed up till now is almost at expert diagnosis level. In addition, many data that have not yet been analyzed can be discovered and studied with AI diagnosis, and this can accurately and quickly diagnose and improve the quality of medical care dramatically.

This research was supported by the project at Institute of Convergence Bio-Health, Dong-A University funded by Busan Institute of S&T Evaluation and Planning (Grant no: 2020).

The authors would like to thank Dr. Adrian Ankiewicz of the Australian National University (Australia) for helpful comments on the manuscript.

The authors have nothing to disclose.

All relevant data are within the paper and its Supporting Information files.

The study was approved by the institutional review board (IRB approval number; DAUHIRB-17-108).