Robot manipulators are growing more widely employed in the retail market, mostly for warehousing, but automating them in-store logistics processes is still a difficult task. Supermarkets and large retail stores face many challenges: shortages, handling, and placement of a single product on shelves. Various issues needed to be considered to develop a robot which can manipulate products of different sizes, shapes, and weight in limited spaces on shelves. The aim of this article is to design and develop a system to address the issues of shortage, identification, moving, and placements of products in supermarkets by properly incorporating database, camera vision, and line following mobile manipulator. A four-wheeled differential drive mobile robot was designed and developed which has a 5 DOF robotic manipulator on it. The line following technique is used to move it around the warehouse. The barcode recognition technique for the localization of product sections and object detection using SIFT is successfully and efficiently employed. The demonstration of the usefulness of the method was shown by carrying out experiments in a relevant environment which imitates a real supermarket.

1. Introduction

The customer experience and comfort have been improved by the implementation of automation techniques at retail shops. A major example is the Amazon Go shops. On the other hand, automated solutions are still limited in the logistics processes of retail stores of the supermarket industry specifically, in Pakistan. The supermarket industry is widely expanding in Pakistan, while being the fact that it still has not achieved the top-notch advancement. There are many areas of advancement and improvement in it following the problems and complications exist in it. One of those problems is the tracking of products on the shelves which are short as well as to put them right in their place on shelves. As the size of supermarkets and number of products increases, it becomes difficult for humans to manage.

Various commercial mobile robots can be used to monitor shelves for inventory management automatically by choosing diverse technologies, such as radio-frequency identification antennas in the Stockbot or the cameras in the Bossa Nova 2020, Tally 3.0, and LoweBot [1]. Another depalletizing-based automated process uses recent artificial intelligence solutions like the Photoneo Depal [2]. Recently, some control solutions based on artificial intelligence approaches have also been presented [3]. A new depalletizer which was able to manage not only boxes but also varied pallets has been proposed in [4].

The inventory management and unloading solutions are already in the market, whereas other in-store automated processes, such as transporting items from the warehouse to the retail store and shelf restocking, can still be taken as emerging technologies. This is a great demand from retailers due to the high costs of in-store logistics, up to 60% of the entire functioning store costs [5]. Fierce market competition has led to a decline in profits and high labor costs make it difficult to ensure that there is enough staff to handle all the work in a store [1].

The shelf refill is the most time-taking task with 50% of such of the time is committed in finding the proper slot on the shelf. Very few research literature is available on this particular automation issue such as in [68]. Most of the research issues are relevant to the convenience store robotic challenge which was launched by the World Robotic Summit [9]. The solutions are mainly based on the usage of vacuum grippers; therefore, they are applicable only in those situations where pick-and-place and grasp poses are similar. Whereas, the shelf refill objective in an actual environment may require cultured skills of manipulation. The REFILLS (robotic enabling fully integrated logistics lines for supermarket) [10] presented a software architecture which implements the shelf replenishment with a comparable speed of human. It also incorporated the KnowRob [11] based task-planning framework.

The current article's contribution is focused on the lower hierarchical level. It comprises of a 7-DOF mobile manipulator prototype that is integrated with a supermarket database system. The proposed system also features a graphical user interface (GUI) for experimentation. A camera vision is also included with the mobile manipulator. As a result of the billing, the supermarket database will be updated. Whenever a product is billed, the system will infer from the database that the product is out of stock. This data will be passed on to the mobile manipulator. After receiving the information, the mobile manipulator will grab the product from the supermarket's warehouse and position it on the shelf where it belongs.

Wireless connection between the database and the mobile manipulator is not used at this time for the sake of simplification. For the purposed system, a laptop will be kept on the mobile manipulator. This laptop will include a database and a graphical user interface. After placing the product in its proper location, the mobile manipulator will also update the database. As a result, the database and the mobile manipulator will be able to communicate in both directions.

1.1. Literature Review

The first steps of any pick-and-place task are object detection and localization. DenseFusion [12] and PoseCNN [13], two of the most current and advanced localization approaches, are nevertheless insufficient for the in-hand maneuvers detailed in this article. The integration of camera vision with a robotic manipulator is a key aspect of our research. Camera vision may be used to identify and regulate the motion of a robotic manipulator, and it can also be used to improve the performance of a robotic manipulator. The proper operation and use of camera vision is fully dependent on determining the point of application. Because determining the point of application, which functions as an input for the system, is required when controlling a robotic manipulator based on its inverse kinematics. However, it is just necessary to specify the settings and specifications for a robot to automatically classify and stow goods [14, 15].

Other than determining the point of application, another component of camera vision is the scanning of barcodes. This domain also has a substantial amount of work [16]. Numerous strategies for barcode detection using camera vision have been developed over the years [17, 18], and many methods and techniques for improving the barcode detection process have been implemented [19].

A great deal of study has also been done in relation to warehouse and inventory management systems and their various components [2029]. The design of the mobile robot and mobile manipulator is critical in this project because proper and error-free design leads to the project's error-free operation. Many recent and old publications [15, 30, 31] propose novel techniques to kinematic modelling and comprehensive design of robot manipulators.

The kinematic analysis of robot manipulators is a topic of debate in a number of classic literature [32, 33]. Peter Corke created a software programme and MATLAB library for kinematic and dynamic design and calculations, which has become the industry standard [34]. Koyuncu and Guzel provided a software package called MSG that is used to test the behavior of robot motion [35]. They established a way for solving the kinematics of the Lynx 6d of Robot and proposed a method for solving the kinematics of the Lynx 6d of Robot. To address the kinematics of the AL5B Robot arm, Qassem et al. proposed a software package [36]. More analysis have been achieved for modeling a 6DOF robotic manipulators using the MATLAB software for the purpose of their design, simulation, and analysis by Iqbal et al., Kumar et al., and Singh et al. [3739].

2. Methodology

To solve abovementioned problems, we developed a system which incorporates the basic components of the system which are mobile manipulator, microcontroller, database, and GUI in a way to timely track down the shortage of products on shelves. Database carries the information about all the products, their barcodes, their optimum quantity, and their actual quantity in warehouse as well as in supermarkets. The database management system is connected to the billing system in practice. However, for the purpose of simplicity, we developed a GUI in place of billing system. The GUI will be used to reduce the number of products from the database. The reduction in the number of products from GUI, then from database will result in the prompt tracking of short products from shelves. It means the system will no sooner have information of shortage of a product than the billing of that product. However, this is a part of usual supermarket management nowadays.

After the information of shortage, the next step is to assess the availability of that product in the warehouse. It has been mentioned above that database also includes information of barcode all products and the quantity of that product in warehouse. It should be kept in mind that, the shortage here does not imply that the products are also unavailable in the warehouse. Database will check if the product is available in the warehouse. If its checking results are positive, the system will communicate to the mobile manipulator. Mobile manipulator, which is included by a microcontroller, will be informed by the information of short products and their barcodes.

The locations of different products with different barcodes are known in the warehouse as well as in supermarket by microcontroller. Microcontroller will travel toward that product in the warehouse by tracing down its location. 5 DOF robotic manipulator will pick up that product. The mobile robot will move toward the location of that product in the supermarket. Robotic manipulator will then put that product right on its place on the shelf.

Here, it must be clear that the line following, barcode scanning, and camera vision techniques will be used for mobile robot path planning, section allocation, and product point determination, respectively.

The mobile manipulator will also provide feedback to the microcontroller in the form of feedback sensors. Moreover, after the microcontroller has completed the task of picking and placing the product in its actual place, it will be required to give this information back to PC, i.e., database. In this way microcontroller and mobile manipulator will have mutual communication. Similarly, microcontroller and PC will have mutual communication. Below, Figure 1 represents the block diagram of proposed methodology, and Figure 2 shows the complete 3D model of robotic manipulator.

2.1. Design Calculations
2.1.1. CAD Model
2.1.2. Design of the Mobile Robot

A mobile robot is a four-wheeled vehicle with differential drive. All considerations are made in calculations according to differential drive principles. Design calculations of mobile robot include the velocity and torque calculations of motor, choice of drive mechanism for mobile robot, and its calculations and choice of motors according to above calculations.

2.1.3. Velocity Calculations

Velocity calculations are the calculations of motor angular velocity in context of desired velocity of mobile robot, which is as follows:

We know that,where, ω is the required angular velocity of motor

After unit conversion to RPM:

It means that both motors of mobile robot should have an angular velocity of 240 RPM or greater.

2.1.4. Torque Calculations for the Mobile Base

In this section, torque requirement of motor will be calculated. For that, we will use the techniques of statics. The first step is to draw a free body diagram.

It can be seen in the free body diagram Figure 3 that the weight of whole mobile manipulator, camera, and laptop is assumed at the center. Due to existence of two motors, these forces are multiplied by 2. Now, we will apply conditions of static systems to evaluate forces:

From (16), we have the following:

Putting it in (15) results in the below:

Which means that both normal forces are equal, i.e.,

We know the radius of wheel, so we can find the torque of motor required as follows:

It means that both motors must have a torque of at least 1.47 Nm.

2.1.5. Selection of Motors

The choice of motor of the mobile robot is very important in design calculation due to the fact that incorrect choice can lead a great loss in term of cost and performance. We are using DC gear motors due to following reasons:(1)DC motors are available with required torque and speed parameters.(2)DC motors are preferred over encoded DC motors due to the fact that we do not require high accuracy because we are using line following technique.(3)DC gear motors are good for high load and torque applications.

2.1.6. Drive Mechanism

Drive mechanism allows for the proper transmission of power from a powered wheel to a powerless wheel. There are three types of drive mechanisms commonly used:(1)Belt drives(2)Chain drives(3)Gear drives

We are using chain mechanism due to following reasons:(1)Chain drive offers zero slip unlike belt and gear drives.(2)Chain drives are more stable than belt and gear drives.(3)Chain drives are highly efficient in power transmission.(4)Chain drives are compact in size and easy to install.

2.1.7. Design of a Robotic Manipulator

We are using a 5 DOF robotic arm in our project. All of its joints are revolute. Therefore, it can be said that it is kind of an articulated manipulator.

2.1.8. Torque Calculations

In the articulated manipulator, the torques of the joints having rotations about x-axis are important to find. Due to the downward act of gravitational force (weight) of the links, the links with the rotation about x-axis will have some natural torque. In order to resist the downward force acting on the links plus to lift the payload, the required torques for the motors will be calculated.

To calculate the torques of the motors, the worst condition is considered, i.e., the robotic arm is considered to be placed horizontal. Now, the torque required to lift the corresponding link with load will be maximum as shown in Figure 4.

From the figure, we are only interested to find the maximum torques about points S, E, and R, respectively. Where ,and are the masses of the links at different points.

Using second condition of equilibrium, required torque of motor about point ‘S’ is as follows:

Similarly, required torque of motor about point ‘E’ is as follows:

For elbow, required torque of motor about point ‘R’ is as follows:

2.1.9. Forward Kinematics

The labeled kinematic diagram of robotic manipulator is shown in Figure 5. The forward kinematics calculations start with the labeling of kinematic diagram according to DH method, basic rules of which are as follows:(i)Direction of z-axis is toward of the joint axis.(ii)x-axis is parallel to the common normal.

As shown in Table 1, there are four DH-parameters a, d . These parameters are designated according to following rules:(i) is the angle from xi-1 to xi along zi−1; For revolute joint is variable.(ii)di is the distance from the intersection of zi−1 with xi to the origin of (i−1) system of axes; For prismatic joint di variable.(iii)ai is the shortcut between zi−1 and zi.(iv) is the angle from zi−1 to zi along xi.

The general transformation matrix is as follows:

Gripper’s position is defined by the p5 vector while orientation defined by x5, y5, z5 vectors provided by the forward kinematics:

2.1.10. Inverse Kinematics

Solution to the inverse kinematics problem is reduced to the calculation of the arguments , , , , based on the gripper’s position and orientation. These six equations in five unknowns may have no solution, though it is possible to consider it for some instances [40]. Equations (16)–(19) can be rewritten in the following forms:

can be evaluated using (29) and (30) straightforwardly:

To evaluate and , we used the geometric method:where values of and can be evaluated as follows:where

From (28),

2.1.11. Work Flow Diagram

As mentioned in previous sections, all important components and aspects of the project were made functional. Line follower robot, 5 DOF robotic manipulator, camera vision, GUI, and various peripheral components of these components were significant components of the project. Important portions and systems of the project, such as barcode identification, object detection, and inverse kinematics, were combined with these components. The project was made to function according to the flow diagram illustrated in Figure 6, after all project components and systems were integrated. This flow diagram depicts the project's whole operation after it has been integrated.

3. Experiments and Results

3.1. Development Stages and Final Stage

The development stages and final stage of the proposed model are given in Figures 7 and 8, respectively.

3.2. Implementation of the Barcode Recognition Technique

For section tracking of warehouse and supermarket, we used barcodes as a visual marker. These markers guided the robot toward the location of items. Each section had its different barcode on its front. OpenCV library was used to access the camera. Barcode and QRcode detection and decoding were done by using the pyzbar library with OpenCV. It first reads the image using cv2.imread(). After reading the image, it decodes the image using pyzbar.decode() and extracts the bounding box (x, y) coordinates from the barcode data, enabling us to localize and determine where the barcode in the input image is. It draws a rectangle around the barcode with OpenCV rect function and also puts a decoded text on the image or scene. Our mobile base robotic arm operates on Raspberry Pi B3. We implemented this technique on Raspberry Pi by using a Pi camera. The flowchart in Figure 9 shows the methodology of a barcode recognition technique and results of barcode recognition are shown in Figure 10.

3.3. Object Detection Using SIFT

To overcome the shortage challenges of products in supermarket, we used a vision assisted robotic arm with mobile base. Vision is incorporated with a robotic arm to pick objects from the warehouse and place them in their respective shelves in the supermarket. To detect and recognize different object detection algorithm like SIFT, SURF, and ORB can be used. Scale invariant feature transform (SIFT) is used in our project because it is robust and invariant to scaling and lightning conditions. The flowchart diagram of object detection using SIFT is shown in Figure 11.

To implement the SIFT algorithm in our project, we used openCV version 3.4.2. It is a built in detector in openCV. First, we have to construct a SIFT object using cv.xfeatures2d.SIFT_create(). After this, SIFT will detect key points of target object. OpenCV provides cv2.compute() which computes the descriptors from the key points. We have key points and descriptors of the target image.

We repeat the same process for the scene image. We got the descriptor vectors of both scene image and target image. Now, we match these two vectors. The minimum Euclidean distance between these two 128 dimensional descriptor vectors will be the best match. For matching brute force and fast library approximation nearest neighbor (FLANN) can be used. The object detection using SIFT before and after center point calculations are given in Figures 12 and 13,respectively.

4. Conclusions

The fixed manipulators on the industry floor have already been developed and established decades ago with great efficiency. The proposed project provided a mobile manipulator which solve many problems faced by today’s supermarket and inventory industry. In time placements of products in supermarkets were handled by properly incorporating database, camera vision, mobile manipulator, and Raspberry Pi (Controller). The four-wheeled differential drive mobile robot was designed with line following capability covered the factory floor area by moving accurately around the warehouse. A 5 DOF robotic manipulator could pick the object effectively with the help of SIFT feature extraction and center point calculation of the object. The barcode recognition technique enabled the identification of product to be placed at right sections in the warehouse. It was done using PYZBAR.

In spite of its good results, many advanced features and desired components could not be introduced due to financial and time limitations.

The future recommendations will likewise focus on addressing the issues. To eliminate interaction with humans on the ground, a line follower robot could be replaced by an overhead transit system. It will significantly increase the speed of the entire operation. There are a variety of additional ways that can be used in this situation. Redevelopment of the product localization process can be done in the future. We urge that additional tests be conducted by replacing the line follower with a SLAM-based navigation system, as we feel that using SLAM-based navigation on a mobile robot can open up new possibilities for the project’s functionality.

RFID tags have proven their reliability in the supermarket and product handling applications [41], and they can be used here instead of barcodes to greatly improve results. To improve results, the model of a mobile robot and robotic manipulator can be changed in terms of materials, shape, joints, and actuators. Picking objects in a cluttered environment will be easier with the learning of a robotic manipulator utilizing a neural network. It is the most essential of all future recommendations because it influences the project’s main goal.

Aside from these design changes, a significant step forward would be to fully automate this semiautomated system. For example, we used a GUI to perform billing actions in this prototype, but the actual system should be connected to a billing server or database and automatically take inputs from the database. In the future, it is recommended that mobile robots be entirely automated.

Data Availability

There are no explicit data used in the proposed research. All the data are available in the paper.

Conflicts of Interest

The authors declare that they have no conflicts of interest.