Comparison of modern nonlinear multichannel filtering techniques using recent full-reference image quality assessment methods

In the paper the quality analysis of some modern nonlinear color image filtering methods is presented. Traditionally, many image filtering algorithms are analyzed using classical image quality assessment metrics, mainly based on the Mean Square Error (MSE). However, they are all poorly correlated with subjective evaluation of images performed by observers. Due to necessity of better image quality estimation, some other methods have been recently proposed. They are especially useful for development of new lossy image compression algorithms, as well as evaluation of images obtained after applying some image processing algorithms e.g. filtering methods. Most of image quality algorithms are based on the comparison of similarity between two images: the original (reference) one and the second one which is processed e.g. contaminated by noise, filtered or lossily compressed. Such a group of full-reference methods is actually the only existing universal solution for automatic image quality assessment. There are also some "blind" (no-reference) algorithms but they are “specialized” for some kinds of distortions e.g. blocky effects in the JPEG compressed images. The last years’ state-of-the-art full-reference metrics are Structural Similarity (SSIM) and M-SVD based on the Singular Value Decomposition of two images' respective blocks. Another important aspect of color image quality assessment is the way the color information is utilized in the quality metric. The authors of two analyzed metrics generally do not consider the effects of using color information at all or limit the usage of their metrics to luminance information in YUV color model only so in this article the solutions based on RGB and CIE LAB models are compared. In paper the results of quality assessment using the SSIM and M-SVD methods some modern median-based filters and Distance-Directional Filter for color images are presented with comparison to those obtained using classical metrics as the verification of their usefulness. comparison to their application using RGB model. Relatively good results obtained using conditional ordering approach in the RGB color space for images without noise or with achromatic noise are caused by the specific character of the presented image with many small details (often very dark and bright).


Introduction
Median filtering is one of the most popular nonlinear techniques of noise elimination in digital images, especially useful for impulse noise removal. One of the most important features related to such an algorithm for grayscale images is the fact that, unlike in linear low-pass filters where the resulting pixel's luminance is calculated as the weighted average of luminance of some pixels from the neighbourhood (within the filter mask), the median filter always chooses one of the luminance levels existing in the pixel's neighbourhood as the result. As the consequence of such choice, there is a guarantee that no new luminance levels are introduced into the image which is especially important for images with limited palette.
Median filtering algorithms for color images also should prevent the situation when new colors, treated as combinations of values for three channels, are introduced into the image. For that reason, independent filtering of each color channel (red, green and blue typically) should not be used. Instead of that some vector based median filtering algorithms have been proposed but their performance is typically compared to existing solutions using conventional image quality metrics [1], such as Peak Signal to Noise Ratio (PSNR) or Mean Absolute Error (MAE). For color images there is a possibility of using Normalized Color Difference (NCD) metric defined in the CIE LAB color space but in many publications their authors use only classical quality measures which are poorly correlated with the way humans perceive images.
However, in recent years some new image quality assessment methods have been proposed, much better correlated with Human Visual System, so there is a need for verification of usefulness of some multichannel filtering methods from the point of view based on some modern image quality metrics.

Multichannel median filtering algorithms
The simplest approach for color median filtering is independent filtering of each color channel (marginal ordering) which can lead to many artifacts in the resulting image and introduce new colors into the image. Regardless of the disagreement with one of the most important aspects of median filtration, this effect can lead to poor results of quality assessment because of potential discolorations caused by ignoring the correlation among color channels. The only reasonable usage can be related to the situations when channels are strongly decorrelated (e.g. so called "television" color spaces such as YUV and YIQ).
The proper methodology of color image filtration is using vector-based filters which utilize data related to all channels. Assuming the most popular RGB color space for each pixel values from each channel are stored in the three-element vector and the ordering criterion of vectors corresponding to the pixels within the current mask is dependent on the particular filter. The simplest approach is just the choice of specified channel as the main one and the values from other channels are analyzed only if two or more pixels have the same value of the main channel. Such approach is known as conditional ordering.
In reduced (aggregate) ordering techniques, the ordering criterion is calculated on the basis of all elements of the vector. The main difference between grayscale and such kind of vector processing is the choice of the output element. In the grayscale median filter (and in the simplest conditional ordering filters) the middle element of the sorted vector is chosen as the result but in vector processing pixels are usually ordered by the criterion related to e.g. aggregated distance to other elements in the current mask (window) so the lowest ranked element is chosen as the result. In these algorithms pixels significantly changing from their neighborhood ("outliers") are usually high ranked and have small chances to be chosen.
Two major groups of reduced ordering filters are Vector Median Filters (VMF) and Vector Directional Filters (VDF). In the first approach the pixel with the minimum aggregated distance to the others within the sliding window is chosen as the result and typical distance measure is Minkowski metric [2]. One of the extensions of such filter is the combination with the linear low-pass filter (Arithmetic Mean Filter -AMF) known as Extended Vector Median Filter [3], useful for elimination of non-impulse noise e.g. Gaussian one. The other well known combination of AMF and VMF is α-trimmed VMF where α elements with the lowest rank in VMF are averaged by AMF [4]. However, such approach can be used for mixed noise and does not guarantee that no new colors will be introduced into the resulting image. Similarly to grayscale weighted median filters such approach is also present in the field of color image processing. The most popular filter is known as Central Weighted Vector Median Filter [5].
The simplest filter from the VDF class is Basic Vector Directional Filter (BVDF) [6] eliminating the pixels with atypical directions in the specified color space (not necessarily RGB). The angular distance can be calculated using the following formula: where x j and x k are the three-element vectors in the specified color space and N denotes the number of pixels inside the processing window. As the result, the pixel with the minimum angular distance A k is chosen. The modification of such algorithm known as Generalized Vector Directional Filter (GVDF) utilizes the cascading structure where the number of BVDF outputs with the minimum angular distance is filtered using e.g. distance norm in the second stage. A combination of median and directional approaches described above is the Distance Directional Filter (DDF) which uses the weighted average of both magnitude and direction defined as [7] ( ) where d(x k ,x j ) is calculated using the Minkowski metric (usually Euclidean distance) as in VMF. Another interesting idea is using HSV color space with conditional ordering as proposed in paper [8]. Such filter, denoted as VMED, is based on the ordering primarily using V component in the ascending order, and then (only for equal values of V) in the descending ordering by S and finally, in the case of equal values of V and S, a hue component is used for sorting in the ascending order. As the result, the middle element is chosen similarly to the grayscale filter.
The filters described above have a number of further modifications, especially towards adaptive algorithms, but generally, the results presented by their authors are based usually only on the classical image quality assessment methods and NCD metric. In this paper, the results obtained by some of the discussed filters are verified using the modern image quality metrics in the CIE LAB color space as well as using the RGB model.

Modern image quality assessment methods
There are two general approaches for image quality assessment -objective and subjective methods. Only the first can be successfully applied in computer systems and for optimization tasks because of the necessity of time consuming human interaction in the methods based on subjective evaluation. A typical approach for subjective quality assessment is using Mean Opinion Score (MOS) as the metric calculated by analysis of the quality scores given by a number of observers. Certainly, such approach can be helpful for verification of existing algorithms but during research with the required direct usage of quality metric, it should be treated as rather a useless solution.
Objective image quality assessment methods are based on the calculation of preferably single scalar value corresponding to the overall quality of the image. More sophisticated algorithms allow the creation of quality maps or may give the opportunity to calculate a vector measure consisting of some independent metrics related to some specific distortions. The only limitation of such approach is the necessity of componing the distorted image with the original one which should be known (full-reference methods). However, existing blind (noreference) or reduced-reference image quality measures are usually sensitive only to specific types of distortions (e.g. block effects on JPEG compression) and their universality is far from the requirements of image processing society. Currently only full-reference objective image quality assessment methods, especially scalar ones, are treated as universal solution and can be utilized in many areas of digital image processing e.g. development of new filtration algorithms, lossy compression methods or video transmission techniques.
The classical measures of that type are based on Mean Square Error (MSE), e.g. Peak Signal to Noise Ratio (PSNR) or similar metrics like Mean Absolute Error (MAE). Most of them, unfortunately poorly correlated with Human Visual System, are briefly described in [1].
The first modern image quality metric is the Universal Image Quality Index [9] proposed by Wang and Bovik in 2002, further extended into Structural Similarity [10] defined as: where X and X' stand for the original and distorted images The constants C 1 and C 2 are the most relevant extension of the Universal Image Quality Index preventing the stability of obtained results (possible division by zero especially for large flat and dark regions) and should be chosen as e.g. C 1 = (0.01 × 255) 2 and C 2 = (0.03 × 255) 2 as suggested by the authors of the paper [10] because they should not introduce any significant changes of the results. X and ' X are the average values and σ is the standard deviation in the original and distorted image blocks respectively.
Using the SSIM index with the sliding window approach, it is possible to create quality map of the image. The default window is 11×11 pixels Gaussian one. The mean value of SSIM index from the whole quality map is treated as the overall image quality metric, sensitive to the three common types of distortions: loss of correlation, luminance distortion and loss of contrast.
Another interesting approach to objective image quality assessment has been presented in the paper [11] and is based on Singular Value Decomposition (SVD) calculated for 8×8 pixels blocks of original and distorted images. As the comparison of such values for each block, the following value is calculated: where s k and s' k stand for the singular values for original and distorted image blocks.
The overall quality metric is calculated using the following formula: where K is the total number of blocks in the whole image and D mid denotes the middle element of the sorted vector D (in fact, the median value).
Most image quality metrics are based on the assumption that evaluated images are grayscale ones. The same situation takes place for the SSIM and M-SVD measures. Their authors argue that color information is not important in image quality assessment at all or using simple conversion of color images from RGB to YUV color space in order to use only the luminance channel (Y) for quality evaluation. However, such approach is rather useless for color image processing, including multichannel median filtering techniques.
On the other hand, there is only one worth noticing quality metric defined particularly for evaluation of color images utilizing the CIE LAB color model (in some applications CIE LUV model is also used), known as Normalized Color Difference (NCD) defined as: where L,a,b denote the CIE LAB channels of the original image and L',a',b' stand for the same for the distorted one. The conversion from RGB to CIE XYZ and then to CIE LAB color space is based on nonlinear functions depending also on the white point [12].
10% achromatic impulse 20% achromatic impulse 20% achromatic mixed 10% RGB impulse 20% RGB impulse 20% RGB mixed Although the NCD measure is based on the perceptually uniform character of CIE LAB color space, it is sensitive only to the color preservation, so its usefulness for quality assessment of noised and filtered images is limited.
As the method potentially useful for verification of some nonlinear filtering algorithms, the usage of SSIM and M-SVD measures in CIE LAB color space (also compared with the application of RGB) instead of often used traditional metrics such as MAE or PSNR is investigated. Test results have been obtained for several commonly used color test images contaminated by achromatic and color impulse noise with various strengths (interpreted as the number of contaminated pixels) filtered using some of the most popular color median filters.

Discussion of results
In the experiments several typical images have been used. Chosen multichannel filters have been applied for images without any noise and contaminated by achromatic and RGB impulse noise as well as by mixed achromatic (Gaussian and impulse) noise. The results obtained for intentionally chosen example image known as "Mandrill" or "Baboon" characterized by a large number of details are presented in Tables 1-7 (PSNR values are expressed in dB). The filters denoted as C-R, C-G and C-B are based on conditional ordering using specified channel as the main criterion. Gaussian noise has been added to the images assuming the mean value equal to zero and standard deviation equal to 50. Original image and its contaminated versions are presented in Fig. 1  comparison to their application using RGB model. Relatively good results obtained using conditional ordering approach in the RGB color space for images without noise or with achromatic noise are caused by the specific character of the presented image with many small details (often very dark and bright).  Usage of CIE LAB color space causes the necessity of modification of typical ranges for each channel, because dynamic range of L channel is from 0 to 100, and for A and B chrominances the range is from -120 to 120. The assumption Pobrane z czasopisma Annales AI-Informatica http://ai.annales.umcs.pl Data: 04/06/2022 00:41:51 U M C S of 256 levels for each RGB channel cannot be made for the CIE LAB model so it affects the obtained results of SSIM and M-SVD metrics in that color space. It is clearly visible for the M-SVD measure where much better quality can be expected analyzing the results obtained for the CIE LAB model in comparison to the assessment of the same images in RGB color space. Nevertheless, one should avoid the direct comparison of results obtained for different color models. The presented results show that there is a serious danger of incorrect conclusions in such a case.
Analyzing the obtained results it can be observed that application of CIE LAB color space for modern image quality metrics such as SSIM and M-SVD does not lead to better correlation with the human perception of images. As the illustration of that problem, the images presented in Fig. 2 can be investigated, especially for the SSIM measure as more attractive because of its much lower computational cost.
The result of filtration using the conditional -red filter presented in Fig. 2a is noticeably worse quality of the image while SSIM calculated in LAB color space increases from 0.3288 to 0.4957 (SSIM values in RGB are 0.4007 and 0.3469 respectively as shown in Table 7). The images presented in Fig. 2b illustrate the effect of using DDF filter for 20% achromatic mixed noise, the result of filtration is perceptually better but SSIM for CIE LAB gives almost identical results (0.6130 after filtration and 0.6261 for noisy image). Fig. 2c illustrates the fragments of original image and that obtained using the conditional-blue filter. The resulting image has noticeably worse quality but the results of NCD and SSIM for CIE LAB as well as PSNR and M-SVD do not illustrate this effect adequately. It can be treated also as the example of M-SVD measure imperfection since its values indicate much better quality after filtration. In that case, only MAE and SSIM calculated on the RGB model illustrate the loss of quality properly, but one should notice that Mean Absolute Error can often produce unexpected results for other types of distortions.

Conclusions
The presented results can be treated both as the verification of usefulness of some filters and the differences between them as well as the confirmation of suitability of some modern image quality assessment methods for evaluation of multichannel nonlinear filters. The methods presented in the paper can be a useful tool for development of color image processing algorithms. It is worth noticing that SSIM is generally more convenient metric than M-SVD because of its lower computational complexity regardless of the fact that calculations are performed using the sliding window approach, not for the blocks. Application of modern image quality metrics for comparison of multichannel median filtering algorithms usually leads to similar conclusions to the usage of classical methods but prevents some errors caused by poor correlation of traditional measures. Taking into account their correlation with the subjective evaluation, modern metrics can be treated as an effective alternative for traditional measures, especially for applications where several classical metrics should be used for reliable comparison of the obtained effects, such as in multichannel nonlinear image filtering methods. In those cases the usage of single modern quality measure should be enough for reliable comparison. Nevertheless, the application of some other color spaces for image quality Pobrane z czasopisma Annales AI-Informatica http://ai.annales.umcs.pl Data: 04/06/2022 00:41:51 U M C S assessment, also for evaluation of some adaptive multichannel median filters, can be an interesting field of further research.