DYNA

1. Introduction

Traditional optical systems are limited by focus range, which means that not all objects in a scene appear clearly defined [1], so only the objects within the field of depth of the camera are focused and are perceived clearly, while the rest of the scene is blurred [2]. Thus, to generate an image with all objects adequately focused, the academic community used the idea of forming a synthetic image by fusing a set of images of the same scene. This is called Multi-Focus Image Fusion (MFIF). The MFIF process consists of merging multiple images with different focal planes to generate an image in which all objects appear sharp [3] without the introduction of any artifacts.

The application of MFIF is widespread and used to solve problems related to three-dimensional reconstruction [2,4], mobile image processing, microscopic imaging [5], and computer vision [6], among others.

To generate an extended-focus image from a sequence of partially focused images of the same scene, digital image descriptors called focus measures [7] are used in different approaches.

MFIF is generally divided into two major types of methods [1,8-9]:

• spatial-domain image fusion methods;

• frequency-domain image fusion methods.

The spatial-domain methods use measurable characteristics of spatial information of image pixels. They estimate these measures pixel by pixel or by using pixel sets [10,11]. The principal advantages of these methods are that they are easy to implement and require low computational complexity. However, they do require rich texture information in an image to generate good results and are usually weak in smooth image regions. Furthermore, grouping of pixels presents difficulties related to the correct determination of group size for quality maximization and to the presence of artifacts generated on border blocks [3].

Various approaches within the special-domain methods of MFIF have been applied. For example, K.L. Hua et al. [11] used random walks on graphs created from several feature sets of focus measures and color consistency to model local and global characteristics. This approach estimated the measures in each pixel locally for each input image and used them to maximize the global focus score and color consistency. On the other hand, a method based on sparse feature matrix decomposition using morphological filtering to extract salient features of original input images, was proposed in [12]. They used a pixel-wise methodology to fuse each sparse feature matrix estimation based on morphological filtering to generate the fused image.

Some works use hierarchical structures called QuadTree for recursively partitioning the pixel space of the image and decompose the input images into blocks of variable size [13,14]. Using a focus measure based on the sum-modified-Laplacian (SML), the method detects the focused regions. So, the resulting image is generated using the focused regions of input images. The main problem with partitioned methods is determination of the correct block size. Avoiding defocused regions in large regions or small blocks with low-contrast variance is the main challenge of QuadTree [14]. Similarly, the accuracy of the partitioning depends on searching deep in the tree, which has a direct impact on the computational cost. Other graph-based works are found in [15,16].

Another approach for dealing with the selection of focused regions in images is the segmentation approach. S. Li et al. [17] proposed a method that uses morphological filtering for rough segmentation of the images based on an initially estimated focus map. The method then uses the image matting technique to refine the segmentation results and a merging process to generate the final image. Another segmentation approach was proposed by M. Nejati et al. [18], based on a training and a testing phase. The training phase constructs a dictionary using focus information maps from local patches of source images. Each pixel from each input image is classified as in focus or not in focus. The final image results from pixel fusion according to a decision map that indicates which source image must be used to obtain the pixel intensity value. Overall, although the segmentation/optimization-based methods applied to the problem of image fusion generate adequate results, they involve a high computational cost.

Frequency-domain image fusion methods transform input images into a frequency domain representation where they are combined. One approach within this category is based on multi-scale decomposition. This is the most commonly reported approach to MFIF. For example, a shift and rotation invariant pyramid representation called Steerable Pyramid was applied by Z. Lin et al. [22]. Other frequency representations reported are Discrete Wavelet Transform [7,23] and Robust Principal Component Analysis [6]. Frequency-domain image fusion methods can be applied to multi-focus and multi-modal images with acceptable behavior. However, they add some noise level and cannot guarantee fidelity of the input image in the final image.

The MFIF problem can be stated as a classification problem where the classifier must decide in which source image the pixel or region has a high focus measure. J. Saeedi and K. Faez proposed a wavelet-based MFIF method which used a two-class Fisher classifier to group the regions into focused and defocused ones [19]. To reduce the number of misclassified regions due to uncertainty, they included a fuzzy logic algorithm. On the other hand, some works reported the use of other types of classifiers, such as neural networks, to tackle the MFIF problem [20,21].

The aim of this study is to compare various classification approaches to the MFIF problem. We selected four popular focus measures and proposed a new one based on morphological features; then we used these as classifier inputs.

This paper is organized as follows. In Section 2, we present the methodology used in this study, including the data and vector feature selection, training, and fusion stages. Section 3 presents the results obtained and comparisons. Finally, Section 4 concludes with a summary and future works.

2. Methodology

We establish the MFIF as a classification problem where the aim is to process a pair of input images, labeling their regions as focused and defocused, in order to build up a final image that merges the focused regions. The methodology begins by selecting the image set used for the training and testing stages. After the image set was defined, we selected focus measures reported in the literature and machine-learning-based classifiers. The training process and the image fusion were the next methodological stages. Finally, we carried out an evaluation step to compare the behaviors of the selected classifiers. Figure 1 shows a block diagram of the methodology used.

Figure 1: Blocks diagram of used methodology.

Source: The authors.

2.1. Image-set selection and rectangular segmentation

Our initial set of images consisted of 30 pairs of multifocus images taken from public datasets [18,24] and a set of images of our own acquired in the laboratory. Every pair consisted of two images: a near-focused one and a far-focused one. Then, using rectangular cropping, a subset of 830 images was constructed and associated with a binary tag (focused:1, defocused:0) by human judgment. Figure 2 illustrates this process.

Figure 2: The scheme used to form the subset of images.

2.2. Focus measures and feature vector formation

In the MFIF context, a focus image operator is a local metric that quantifies the quality of focus in an image region. Ideally, when the region is perfectly focused, these operators must generate a maximum value [25] that decreases in a similar way when the image becomes blurred. Many focus metrics have been proposed by the scientific community. A typical focus metric should satisfy the following requirements [7]:

• independence of image content

• monotonicity of blur

• unimodality (only one maximum value)

• value variation according to degree of blurring

• minimal computation complexity

• robustness to noise

We selected four focus measures frequently used in reported works [7] and proposed a new focus measure based on morphological features.

Let us consider f 𝑥,𝑦 , the intensity level of a pixel 𝑥,𝑦 . The selected focus measures are defined below:

2.2.1. Energy of Laplacian

The Energy of Laplacian (EOL) of an image 𝑓 is computed as:

Where

2.2.2. Sum-modified Laplacian

The modified Laplacian is a proposal to avoid the cancellation trend of the second derivate in the EOL basic definition [25]. So, the SML is defined by equations (3) and (4):

where 𝛽 is a spacing parameter addressing the accommodation of texture variation in the image and set to 𝛽=1.

where 𝑇 is a discrimination threshold value and 𝑁 is the window size.

2.2.3. Energy of image gradient

This operator is based on the concept of determining the local high-frequency variations. The Energy of Gradient (EOG) can be computed as:

Where

2.2.4. Spatial Frequency

Spatial frequency (SF) is a modification of the energy of image gradient operator. It is defined as:

where RF and CF are the row and column frequency respectively.

2.2.5. Energy of morphological features

We propose a new focus measure based on the metric defined in [17] and the concept of the energy of the image. We named this focus measure as the Energy of Morphological Features (EMF). We used a combination of bottom-hat and top-hat operations to first extract salient local features and then calculate the sum of maximum values for a window.

Let T and B be top-hat and bottom-hat morphological operations, respectively, centered at the pixel with coordinates (i,j). We describe the EMF of a local window W of an input image I. Finally, we build up a feature vector 𝛾 composed of the ratio of each focus measure of input images A and B. Figure 3 shows the structure of the feature vector.

Figure 3: Vector feature structure.

Source: The authors.

2.5. Fusion scheme

The fusion scheme proposed in this work is composed of two main stages. In the first stage, a binary mask is generated with information about which regions of both images have high focus (Fig. 4). In the second stage, the fused image is obtained using input images 𝐴 and 𝐵 and the binary mask obtained in the first stage (Fig. 5).

Figure 4: Scheme of the generation of binary mask 𝑍 in stage 1.

Source: The authors.

Figure 5: Scheme of stage 2 for the generation of the fused image F.

Source: The authors.

2.5.1. Stage 1 - Binary mask generation

The first stage begins by moving a rectangular window over a pair of input images and iteratively describing each window as a feature vector. Then, the ratio 𝛾 of these two vectors (Fig. 3) is used as an input for a trained classifier which returns a binary label (focused:1, defocused:0) for the highest focus in image 𝐴 or 𝐵 respectively. This binary label is used to build a binary mask 𝑍 of resolution equal to the moving window size. This mask contains information about the highly focused regions in both input images.

Equation (21) describes stage 1, where 𝑍 is the binary mask matrix, the range ( 𝑥 0 : 𝑥 1 , 𝑦 0 : 𝑦 1 ) is the image portion of the sliding window, 𝐶 is the binary classifier, and 𝛾 𝑥 0 : 𝑥 1 , 𝑦 0 : 𝑦 1 is the ratio of the feature vector of input images 𝐴 and 𝐵 for the sliding window in the range ( 𝑥 0 : 𝑥 1 , 𝑦 0 : 𝑦 1 ).

A scheme of the generation of the binary mask 𝑍 in stage 1 is shown in Figure 4.

2.5.2. Stage 2 - Image fusion

It is possible to generate a fused image using the binary mask obtained in stage 1, but because of the rectangular nature of the moving window, a high number of artifacts can be generated in regions containing borders of objects (see the rough behavior of borders and holes in 𝑍 in Figure 4). An improved result can be obtained by smoothing the binary mask.

This stage begins by applying the morphological operation close to group individual pixels and fill holes in the binary mask. Then a low-pass filter (median blur) is applied to delete random noise and smooth the borders of the binary mask. Both operations used rectangular kernels of the same size. Finally, using the smoothed binary mask 𝑍 ′ , input images 𝐴 and 𝐵 are cropped to generate the fused image 𝐹. This process is illustrated in Figure 5.

Equation (22) describes stage 2, where 𝐹 is the image fused using the proposed scheme, 𝐴 and 𝐵 are the input images, 𝑍 𝐴 ′ and 𝑍 𝐴 ′ are the pixels of the smoothed mask belonging to images 𝐴 and 𝐵 respectively, and × is an element-wise multiplication of two matrices.

3. Experiments and results

In this section, we present detailed experimental settings used to evaluate the performance of the classification approach for MFIF from two perspectives: classifier performance and fusion quality. The first evaluates the learning ability of a classifier to decide whether or not an image is focused. The second measures the fusion result generated by a classifier of the information contained in the input images. Finally, we discuss the results exhibited.

3.1. Classifier performance

We selected five binary classifiers and trained them with the parameters shown in Table 1 and a dataset of 830 images labeled as binary for focused or defocused cases.

Table 1: Parameters used for training binary classifiers

Source: The authors

Training parameters were obtained experimentally, selecting the set of parameters that allowed us to obtain the best results.

For each classifier, we progressively changed the dataset size and used the k-fold cross-validation technique [40] with 𝑘=10 in each iteration, to measure the learning and test precision/error. This allowed us to evaluate the performance variation (standard deviation) of each classifier. Table 2 shows the mean and standard deviation of precision of training and testing for each classifier. The highest scores are obtained by k-NN for training precision and by MLP for test precision. In contrast, the lowest precision is obtained by using Naïve Bayes in both the training and test stages.

Table 2: Mean and standard deviation of classifier precision for training and test stages.

Source: The authors.

Learning curves obtained for each classifier are shown in Figure 6. The shaded area around the learning curves indicates the standard deviation of training error for 10 iterations per step.

Figure 6: Learning curves for the binary classifiers used. Red and green lines, respectively, represent the training and cross-validation scores obtained when varying the training sample size.

Source: The authors.

As shown in Figure 6c and Figure 6d, MLP and SVM are the most stable classifiers, as the learning and test curves do not show large differences when the training sample size is varied. MLP and k-NN achieve the highest scores in the training and test stages, but MLP presents high variation when using cross-validation (green shadow).

LDA (Figure 6a) and Naïve Bayes (Figure 6e) have the highest variations in training and test scores, which means that more iterations are required to train a good classifier.

3.2. Fusion quality

A total of 30 pairs (near and far focused) of input images were fused using the proposed scheme. Fused images were evaluated using the fusion quality metrics detailed in Section 2.6. The mean fusion quality for each classifier over 30 cases is shown in Table 3.

Table 3: Average fusion quality obtained with different classification methods

Source: The authors.

The Naïve Bayes classifier obtained the highest quality score for the VIFF and Qabf metrics, and MLP obtained the highest quality score for the FMI metric among all classifiers.

The high mean fusion quality achieved by MLP is expected because this classifier obtained the highest test score, as shown in Table 2. On the contrary, Naïve Bayes achieved the lowest training and test scores (Table 2) out of all classifiers, but the highest fusion quality according to two out of three metrics among all classifiers (Table 3).

Some examples of the fusion results of the classifiers with the best results (Naive Bayes and MLP) are presented in Figure 7. Subtle differences in the masks generated from both classifiers can be observed. A qualitative judgment about this result is that MLP generates fewer artifacts than Naive Bayes.

Figure 7: Results of image fusion scheme proposed for classifiers with the best fusion results: Naive Bayes and MLP. Column 1) input image 1; column 2) input image 2; column 3) Naive Bayes: near focus plane mask; column 4) MLP: near-focus plane mask.

Source: The authors.

Differences in fusion quality can be observed near the borders of objects, where artifacts are present due to the ambiguous nature of focus of the borders of objects in a natural scene (Fig. 8).

Figure 8: Magnified regions of fusion results obtained by a) FFN-MLP; b) k-NN; c) LDA; d) Naïve-Bayes; and e) RBF-SVM.

Source: The authors.

4. Conclusions

The results obtained in the previous section show that the automatic classification concept can satisfactorily address the MFIF problem; that is, a classification scheme can be used to decide whether an image is focused or defocused based on local features. Thus, using this classification scheme, a fused image can be generated in which most regions or pixels are focused. The main contribution of this work is that it compares different classifiers in an MFIF scheme, thus evaluating which one obtains the best results in both learning and fusion stages. However, this work only evaluates the main classifiers found in the literature, and other classifiers not evaluated in this work may obtain better results.

The most important conclusions obtained from this work are as follows:

• Classification precision does not guarantee fusion quality; that is, a classifier can have a low training and testing score but a high fusion quality.

• Naïve Bayes is an example of the latter. It is well known that Naïve Bayes is a robust classifier, and thus, subject to overfitting problems. Thus, it can perform well with small amounts of training data, which is the case found in this work. It is possible that this can be related to the number of artifacts a classifier generates and therefore the fusion quality.

• The MLP classifier shows good results in both the classification stage (training and test) and the fusion stage (fusion quality).

• Artifacts are generated near the borders of objects in the scene due to the spatial nature of the scheme used and the rectangular shape of the moving window. A good classifier generates fewer artifacts in these regions.

• MFIF is not a trivial problem, and a specific classifier can directly affect the fusion quality obtained using the proposed scheme in this work.

• Both the size of the moving window and kernels for morphological operations directly affect the results. Thus, future work should focus on evaluating the impact of these parameters on fusion quality obtained by this scheme.

Furthermore, the scheme proposed in this work could be adapted to address multi-focus scenes that present more than two (near and far) focal depths. An optimization technique based on region-growing could thus be used to generate a better smoothed mask 𝑍 ′ that encloses the objects that appear in the scene more precisely.