Abstract
This paper proposes a visual dirt detection algorithm and a novel adaptive tiling-based selective dirt area coverage scheme for reconfigurable morphology robot. The visual dirt detection technique utilizes a three-layer filtering framework which includes a periodic pattern detection filter, edge detection, and noise filtering to effectively detect and segment out the dirt area from the complex floor backgrounds. Then adaptive tiling-based area coverage scheme has been employed to generate the tetromino morphology to cover the segmented dirt area. The proposed algorithms have been validated in Matlab environment with real captured dirt images and simulated tetrominoes tile set. Experimental results show that the proposed three-stage filtering significantly enhances the dirt detection ratio in the incoming images with complex floors with different backgrounds. Further, the selective dirt area coverage is performed by excluding the already cleaned area from the unclean area on the floor map by incorporating the tiling pattern generated by adaptive tetromino tiling algorithm.
1. Introduction
In the past decade, mobile robot platforms are playing a vital role in the field of security and surveillance, laboratory assistance, medical robots, and professional cleaning tasks. Among them, autonomous cleaning robots are getting more attention and have a highly competitive market due to high demands in homes, hospitals, food courts, and industries. Typically, autonomous cleaning robots have various cleaning modes, such as an auto mode to clean the entire area, selective area cleaning mode for cleaning designated areas, and spot cleaning mode for cleaning a specific spot [1]. Among these three modes, selective area cleaning mode and spot cleaning mode are challenging task, which requires efficient dirt detection mechanism and optimal area coverage planning in the segmented dirt area. Generally, piezoelectric debris sensor or optical sensors are widely used for dirt detection in autonomous cleaning robots [2–4]. However, these sensing devices are able to detect dirt only when it strikes on the sensor and not able to identify dirt density and stains on the floor [2]. Hence, these devices are not able to predetermine the amount of cleaning effort required to achieve better cleaning [2].
Vision-based dirt detection is an emerging technique for autonomous cleaning robots [5–7]. It provides the efficient solution for detection of various kinds of dirt and stains on the floor. Moreover, vision-based systems are passive in nature and do not require to touch the debris for detection physically. It is also possible to detect and estimate the density of debris on the floor by using image processing techniques. Through vision-based dirt detection scheme, autonomous cleaning robots can predetermine dirty and cleaned areas and can adjust the operational mode for achieving efficient cleaning. Furthermore, the robot’s cleaning time can also be reduced by excluding the already cleaned region [2].
Andersen et al. [5] developed a visual cleaning sensor for autonomous cleaning robot based on computer vision and high-power light-emitting diode. The cleaning sensor detects the dirty area and creates the dirt map by analyzing the surface images pixel-by-pixel using the multivariable statistical method. In [6, 7], Bormann et al. adopted a spectral residual image filter for detecting the dust and dirt on a floor image, where maximal filter response and minimal filter response of the different floor pattern are used to fix dirt thresholds. However, it is clearly shown in the experimental results that the detection ratio of the scheme drastically varied based on different surfaces. Vision-based dust and mud detection algorithm is proposed in [8], where maximally stable extremal regions (MSER) and canny edge detection techniques are used to recognize the dust particles on the floor, and then spectral residual filter and RGB filter are used for detection of mud and its color. Grünauer et al. [9, 10] proposed an unsupervised dust spot detection algorithm for industrial cleaning robot where Gaussian mixture model (GMM) is employed on RGB color image for distinguishing the floor and dust spots. Martínez et al. [11] proposed vision-based dirt detection for planar surfaces. The author uses RANSAC algorithm for surface plane detection; then the median filter and thresholding technique are employed to remove the background of the surface and figure out the dirty regions. Machine learning- and pattern recognition-based dirt detection are proposed by Proietti et al. [12]. The authors use the k-NN (k-nearest neighbor) classifier to determine the size and shape of the dirt particles. However, these methods have some practical limitations and drawbacks; the detection ratio is relying heavily on the surface texture and fails to detect the small and tiny dirt particle on the brighter surfaces [6–8].
Another key challenge for floor cleaning robot is deploying the suitable motion planning to cover the dirt-defined areas effectively. Generally, fixed morphology robots use random walk, wall-to-wall, spiral, zigzag, and combined patterns to cover the segmented dirt area [3, 13, 14]. Gylling proposed a greedy heuristic-based selective area coverage for floor cleaning robot where simple forward and backward motion planning is employed to cover the dirty area [13]. In [4], Lee and Banerjee utilized straight pattern for covering the Bayesian framework-based dirt map. Wall-following algorithm is proposed by Hasan et al. for covering the most dust-accumulating region on walls and corners [15]. Hess et al. proposed poisson-driven dirt maps based selective area coverage for floor cleaning robot where random walk pattern is employed to cover the randomly located dirt cells [3]. However, these motion patterns are ideal for covering the blank surface and cannot fit to cover the complex floors with an arbitrary shape having sharp corners and narrow spaces [16, 17].
Reconfigurable morphology robots can cover the maximum area and arbitrary workspace than a fixed morphology robot [18–22]. Prabakaran et al. proposed a novel motion planning technique based on tetromino tiling algorithm for reconfigurable morphology robots hTetro [18]. Here, skew tetrominoes (τ1-τ4) and T-shapes tetrominoes are utilized to generate the clustered tiling patterns for effectively covering the maximum workspace. The same authors have proposed a tromino tiling theory-based motion planning technique for hTromo robot. The algorithm uses L- and I-trominoes pattern for generating the clustered tiling pattern [23]. Though the works mentioned above [18, 23, 24] discuss the entire area coverage using tetromino and tromino tiling algorithms, they do not address the selective dirt area coverage in a given workspace.
In order to solve the dirt map-based selective dirt area coverage problem, this paper presents improved visual dirt detection algorithm using a three-stage filtering technique for enhancing the dirt detection ratio and a novel adaptive tiling algorithm for selective dirt area coverage. The three-layer filtering framework integrates periodic pattern detection filter [25], edge detection filter, and noise filters, where the periodic pattern detection filter is applied to filter out the floor’s regular patterns from the image; then edge and noise filters are employed to detect the dirty regions of the image. Finally, the dirt-mapped regions are fed to the adaptive tetromino tiling algorithm to generate optimal tiling patterns for the segmented dirt area. The techniques presented in this work can be applied for efficient floor cleaning in food courts using reconfigurable morphology robots (for example, with hTetro robot).
The proposed algorithm is tested in MATLAB environment with ACIN [9], Bormann dirt database [7], and real test images. Experimental results confirm that the proposed scheme achieved better dirt detection ratio and enhanced selective dirt area coverage using reconfigurable morphology.
The remainder of this paper is organized as follows: Section 2 explains dirt detection and segmentation framework followed by the polyominoes tiling theory in Section 3. In Section 4, we present the experiments and discussions on the results obtained on datasets and data recorded on different types of floor surfaces with different dirt type. Finally, conclusions and future works are given in Section 5.
2. Proposed Approach for Dirt Detection and Segmentation
In this work, it is assumed that most of the indoor floors exhibit regular patterns which can easily be detected and removed by the image processing techniques. We have applied a three-stage image processing scheme for detection of floor patterns and removing the background from input images, leaving only dirty regions behind. After that, an adaptive tiling scheme is applied to these segmented regions for selective dirt area coverage.
Figure 1 illustrates the functional block diagram of the proposed vision-based selective dirt area coverage scheme. The functional description of each stage in three-stage filtering is given below. The second part consisting of adaptive tiling is described in Section 3.
2.1. Three-Stage Filtering
2.1.1. Periodic Pattern Detection Filter
This step filters out the background of the floor image leaving only the dirty parts. Before starting this step, some preprocessing is performed on the incoming images, e.g., including image plane normalization and grayscale conversion. Typically, in the spatial domain, periodic pattern on the floor will have distinct peaks in the frequency domain. This property has been used to identify the floor pattern and dirt content from captured images, as typical floors exhibit the periodic pattern which is detected in the incoming images of the floor by periodic pattern detection filter. The filter uses the self-filtering technique to compute the appropriate filter function for the identification of floor pattern frequency. It automatically varies the filter function according to floor background pattern. The filter function of the periodic filter has been determined through computing the magnitude of the Fourier-transformed image. Mathematically, the periodic pattern detection filter is described as follows.
Initially, the input image is transformed into frequency domain by applying the 2D fast Fourier transform (Eq. (1)); then the periodic pattern detection filter is employed on the transformed image using Eq. (2). where is the filtered image, is the Fourier-transformed source image, and is the frequency domain filter function. It can be determined by applying FFT shift operation on the Fourier-transformed source image and then choosing the maximum log components, as given in Eqs. (3)–(5) as follows:
After computing the filter function , the periodic patterns are suppressed in frequency image according to Eq. (6).
Finally, the filtered image is transformed back from frequency space to image space with the inverse Fourier transform according to Eq. (7).
2.1.2. Edge Detection and Enhancement
This step performs the edge detection and enhancement on the background subtracted floor images. Due to the high filtering effect on the previous stage, the edges of the dirty part are slightly the blurred which may disturb the visual dirt analysis in later stages. It can be overcome by applying the Sobel edge filter to identify the dirt from blurred region effectively. After identifying the edges, the weak edges are sharpened as shown in Figure 1.
2.1.3. Noise Removal and Dirt Analysis
Once the grayscale image has been enhanced by Sobel filter in stage 2, the grayscale image has been converted into a binary image for dirt detection analysis in a later step. With the binary image, it is easy to differentiate dirty area from the background. The grayscale image is converted to a binary image via thresholding. Figure 1 shows the gray converted binary image at stage 3. The output binary image is surrounded by residual noise. It can be effectively removed by the application of the median filter on the binary image. Figure 1 shows the binary image after implementing the median filter. It is clear that the median filter completely suppresses the residual noise while the filter preserves the dirt regions. After detecting and segmenting the dirty regions on the floor in the current image frame, the output of three-stage filtering is fed to the next stage, which is selective dirt area coverage by adaptive tiling. This step involves finding the suitable clustered patterns for the segmented dirt region.
3. Polyominoes Tiling
The polyominoes theorems are used for selective dirt area coverage in this work. We have introduced general polyominoes tiling theory first. Polyominoes are basically rectilinear blocks arranged by connecting unit squares [26]. Typically, polyominoes can be categorized into unbounded, one-sided, and static polyominoes. It depends on spatial alignment, geometrical mutation, and chirality. For instance, the set domino, created by the blending of two polyomino equal squares, can have single one-sided, single free, and two fixed dominoes as its subgroups. Similarly, triomino has three polyominoes, through which we can be able to build two unbounded polyominoes, two one-sided polyominoes, and six static polyominoes [24]. In the case of tetromino, which consists of quad polyominoes, it can be able to construct five unbounded, seven single sided, and nineteen static tetromino.
3.1. Adaptive Tiling-Based Selective Dirt Area Coverage
This section describes how tetrominoes tiling pattern-based reconfigurable morphology is utilized for effectively covering the segmented dirt area. The detail description of the tetromino tiling algorithm and adaptive tiling scheme for segmented dirt area coverage planning is described as follows.
3.1.1. Tetromino Tiling Theory
The tetromino tiling algorithm generates clustered reconfigurable morphology for efficient area coverage using skew tetrominoes (τ1-τ4) and T-shapes tetrominoes (T1-T4), as illustrated in Figures 2(a) and 2(b).
Case 1. Let and be the sizes of the rectangle to be tiled. The region of the rectangle must be a manifold of 4, (l × w) = 4n, where . It indicates that either should be divisible by 4. Then the rectangular area may be covered through the organization of T-tetrominoes as illustrated in the Figure 3.
Case 2. Let l = 4 and 4 ≤ w ≤ 7, then a rectangular area may be covered through the organization of T-tetrominoes and skew tetrominoes as illustrated in the Figure 4.
Case 3. Let w > 7 and , then w = 4n + r, where is an integer. Similarly, it permits for dividing of (4 × w) rectangular area into n (4 × 4) rectangles and one (4 × ) rectangle. Figure 5 illustrates coverage planning for various sizes of grids according to case 3.
(a)
(b)
3.1.2. Adaptive Tetromino Tiling Algorithm
Adaptive tiling scheme describes the utilization of reconfigurable morphology for effectively covering the segmented dirt area. It utilizes (4 × 4), (4 × 5), (4 × 6), and (4 × 7) clustered tetromino tiling pattern to cover the segmented dirt area. The algorithm automatically groups the segmented dirt area and fix the suitable clustered tetromino tiling pattern(s). For example, to determine the clustered pattern, the captured region is divided into small rectangle blocks as shown in Figure 6. The size of each rectangular block is considered to be equal to a single block of simulated reconfigurable robot. The rectangle box which wholly or partially reside in the dirt map are considered for tiling and are grouped as (4 × 4), (4 × 5), (4 × 6), and (4 × 7) grids. The rectangle boxes outside from the dirt map are considered as cleaned area and excluded from the tile generation. However, some of the outside rectangular boxes adjacent to the dirt boundary may be included in the grouping to fit the clustered tile sets in the dirt map. The adaptive tetromino tiling algorithm is described in Algorithm 1.
| ||||||||||||||||||||||||||||||||||||
In Figure 6, the regions (a, b, c, and d) illustrate how clustered tetromino tile set(s) is fixed automatically according to this algorithm. For example, the region (a) has a size of (4 × 10) which is tiled with two sets of (4 × 4) patterns and one set of (4 × 6) pattern. Similarly, (4 × 12) sized dirt regions (b and c) have been covered with three (4 × 4) clustered pattern. The next region (d) having a size of (4 × 11) is tiled with one (4 × 4) and one (4 × 7) pattern.
3.1.3. Comparison with T-Trominoes
In contrast with t-trominoes [23], the tetromino clustered tile sets can cover the maximum area with less number of reconfigurable shapes. For example, Figure 7 shows the comparative analysis of t-tromino and tetromino tiling pattern-based area coverage for (4 × 6) grid, where t-trominoes required 8 tile sets, whereas tetromino tiling scheme can cover this area with 6 tile sets only.
(a)
(b)
Besides, any grid can be tiled through clustered tetromino pattern of (4 × 4), (4 × 5), (4 × 6) and 4 × 7 sets. But the grids size of (3 × 5), (3 × 7), and (3 × 9) cannot be covered by any arrangements of t-trominoes tile sets [23]. Hence, tetromino patterns have more flexibility in covering the maximum area of the arbitrary shape.
4. Experiments, Results, and Discussion
Two different experiments are performed here for the validation of vision-based adaptive selective dirt area coverage scheme in this work. The first part of experiments is for validating the dirt detection algorithm with real dirt images and images from dirt database [7, 10]. The second one is for validating the adaptive tiling scheme with generated dirt map.
4.1. Dirt Detection Algorithm Validation
The visual dirt detection algorithm is verified with our real dirt images and ACIN [10] and Bormann dirt dataset [7]. Our dirt database is captured at different floor backgrounds with various dirt materials as shown in Figures 8–10. The selection of dirt ranges from a different color of paper snippets, wood waste, fruit skins, flower petals, and food grains and different color of soil and liquid stains. Figures 8–10 shows the detection results obtained from our visual algorithm for different floor images. The detected dirt objects are mapped by visual dirt map as shown in the Figure 8. The map results along with numerical results for dirt detection ratio () in Table 1 show that the proposed scheme gives very low false alarms and achieves higher dirt detection rate.
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
The robustness of the visual dirt detection algorithm is assessed by measuring the dirt detection ratio (α) of different surfaces with various kinds of dirt types. The detection ratio of the algorithm is measured by computing the area of all the ON (true) pixels in a binary image by summing the areas covered all pixels in the image. The area of an individual pixel is determined by looking at its 2-by-2 neighborhood. Table 1 shows the detection ratio of objects for different floor background with different types of dirt.
From Table 1, it is evident that the proposed visual dirt detection algorithm’s detection ratio is stable for different floor types as well as for various kinds of dirt present. The maximum variation is seen for “tea grains” dirt. The reason behind this may be that tea grains are spread randomly just like the mosaic pattern on the floor. The size of tea grains is also comparable to mosaic grains, so detection ratio varies.
4.2. Using Dirt Database
In addition, we verified the dirt detection ratio of the proposed visual algorithm with ACIN [10] and Bormann et al.’s [7] dirt database. The dirt database contains different types of dirt images taken at home and office environments with imbalanced light conditions and high frequent floor textures. The detection results of ACIN and Bormann’s dirt database are shown in Figures 11–16.
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
(a)
(b)
(c)
4.2.1. ACIN Dirt Database
The proposed three-stage filtering is applied to ACIN datasets on different kind of floors with different dirt types (cardboard, liquid stains, and blurred floor with food stains) in Figures 11–13, respectively. The results show that our proposed three-stage filtering segmented the dirt from floor background faithfully. In Figure 14, we used images with varying degree of light conditions and applied the proposed algorithm to detect dirt in the input image. The output of the three-stage filter detects the dirt stains even in varying light conditions.
4.2.2. Bormann Dirt Database
The proposed three-stage filtering is applied to the second dataset given in [7] with dirt type as street dirt and leaves as given in Figures 15 and 16, respectively. The results show that our proposed three-stage filtering segmented both kinds of dirt from floor background faithfully.
4.3. Selective Dirt Area Coverage with Adaptive Tiling
Adaptive tiling-based selective dirt area coverage scheme has been validated with visual dirt maps obtained from two different test beds of the ground size of (400 × 160 cm) as shown in Figures 17 and 18. To obtain the dirt map, various types of dirt are scattered on the test bed and images are captured by AVER Portable Visualizer Module [27]. The visualizer module is comprised of a 5Mega pixel, 180° FOV CMOS camera sensor, four infrared marker to measure the focused area, flexible arm for adjusting the camera height, and camera head tilt from −90 to 90 degrees. To capture a floor area of ground size (400 × 160) cm, the camera height is adjusted to 100 cm and inclined at an angle of 45 degrees downwards to cover the shooting area as shown in Figure 19. Figures 17(a) and 18(a) shows two types of test beds: test bed 1 is a mosaic floor, and the test bed 2 has tiles on it. Figures 17 and 18(b) show the respective dirt map for captured floor images generated by improved dirt detection algorithm described in Section 2.
(a)
(b)
(c)
(a)
(b)
(c)
After obtaining the visual dirt map, the adaptive tiling algorithm has been applied on visual dirt map. Figures 17 and 18(c) illustrate the tetromino tiling-based selective dirt area coverage scheme for segmented dirt map where the dirt region is automatically grouped into a cluster of (4 × 4), (4 × 5), (4 × 7), etc. according to the adaptive tiling algorithm. For instance, in Figure 17(c), the gray areas of the generated map indicate the cleaned regions, and the remaining regions are filled with some dirt. The segmented dirt regions can be covered using the tiling pattern (4 × 5), (4 × 4), and (4 × 7). In Figure 17(c), the fully dirty regions cover 42 numbers of (4 × 4) tiling set (blue), whereas the partially dirty regions cover (4 × 5) tiling (green) arrangement twice and (4 × 7) tiling (yellow) arrangement only one time. In Figure 18(c), tile set (4 × 5) and (4 × 7) are applied twice and (4 × 4) by 29 times. The results achieved through this work demonstrate how the adaptive tiling-based scheme can be applied for maximum coverage of arbitrary-shaped selective dirt area.
4.3.1. Selective Dirt Area Coverage Validation with Varying Distances
To analyze the dirt map-based selective area coverage, the dirt is spilled in 400 × 160 cm test bed region with different distances (e.g., 150 cm and 350 cm) from the visualizer module as shown in Figures 20 and 21. Figures 20, 21(b), and 21(c) show the dirt map generation and adaptive tiling-based dirt area coverage, respectively.
(a)
(b)
(c)
(a)
(b)
(c)
The obtained results reveal that the proposed vision-based dirt detection scheme is able to recognize the dirt surrounded up to 400 cm distance from current position. Similarly, the arbitrary shape of generated dirt region can be covered maximumly through the reconfigurable morphology using adaptive tiling pattern.
4.3.2. Comparison with Existing Schemes
For generating the dirt map, in Hess et al. [3] and Lee and Banerjee [4] scheme, the cleaning devices move around the entire workspace once to collect the dirt information. It needs extra time and energy in traversing the targeted area. Whereas in the proposed scheme, reconfigurable morphology cleaning robot may take images while traversing the target area, and by applying the techniques presented in this work, may detect the dirt and its intensity and apply appropriate cleaning modes. Moreover, the adaptive tiling scheme enhances the maximum dirt area coverage by tiling the constrained regions in the operation area.
5. Conclusion
In this work, we presented a selective dirt area coverage scheme for reconfigurable morphology based on visual dirt detection technique and polyominoes tiling theory. Three-layer filtering framework is used for effective dirt detection and segmentation from complex floor pattern. The experimental results prove that the proposed dirt detection algorithm is highly reliable and robust against imbalanced light conditions and high frequent floor textures and is able to detect the dirt on the unknown surface without any previous knowledge. Furthermore, selective dirt area coverage is maximized through tiling pattern generated by adaptive tetromino tiling algorithm. The integration of visual dirt detection and polyominoes tiling-based area coverage will effectively reduce the floor cleaning efforts by selecting the dirty region through optimal pattern generation. In the future work, we have the plan to implement proposed selective area cleaning strategy to a real reconfigurable robot called hTetro developed in SUTD.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work is financially supported by the National Robotics R&D Program Office, Singapore, under Grant No. RGAST1702, the Singapore University of Technology and Design (SUTD), and the University of Engineering and Technology Lahore (UET), Pakistan, for granting postdoctoral research leave, which are gratefully acknowledged for conducting this research work.