An RGB-D Camera-based Travel Aid for the Blind

Without the mobility capability, no can have an independent life. The blind or the visually impaired people greatly suffer from the independent mobility problem; therefore, the low quality of their life far exceeds our imagination. Nowadays, most of the blind persons still rely on the use of white canes. Only the minority of the blind depends on guide dogs because the training fee and the breeding fee for a fully trained guide dog is usually not affordable to every blind persons. While the white cane has many appealing properties (e.g., light weight, small size, and low cost), its main problem is that it cannot detect obstacles beyond its reach of 1-2m. In addition, a white cane cannot actively point out which way the cane holder may walk forward or remind in advance him or her of some obstacles in front of him or her.


Introduction
Without the mobility capability, no can have an independent life. The blind or the visually impaired people greatly suffer from the independent mobility problem; therefore, the low quality of their life far exceeds our imagination. Nowadays, most of the blind persons still rely on the use of white canes. Only the minority of the blind depends on guide dogs because the training fee and the breeding fee for a fully trained guide dog is usually not affordable to every blind persons. While the white cane has many appealing properties (e.g., light weight, small size, and low cost), its main problem is that it cannot detect obstacles beyond its reach of 1-2m. In addition, a white cane cannot actively point out which way the cane holder may walk forward or remind in advance him or her of some obstacles in front of him or her.
Fortunately, with the advance of the modern information and communication technology (ICT), many different kinds of electronic travel aids (ETAs) have been developed and some of them were even commercialized. The goal of ETAs is to help the blind people perceive environments surrounding them. While some ETAs are based on the use of sensors (e.g., ultrasonic sensors, laser beams, visions), some of them are based on robotics [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18]. For example, a dynamic ultrasonic ranging system as a mobility aid was introduced by Gao & Li [1] and a DSP-based ultrasonic navigation aid was proposed by Mihajlik et al. [5]. The Guide Cane based on mobile robot technologies was developed by Ulrich & Borenstein [6] and a mobile robotic base with an RFID reader and a laser range finder was introduced by Kulyukin et al. [9]. A vision-based travel aid was discussed by Su et al. [15]. A portable indoor localization aid incorporated with a laser scanner was introduced by Hesch & Roumeliotis [16]. A co-robotic cane which uses a 3-D camera was developed by Ye et al. [17]. Each has its own merits, limitations, and disadvantages. For example, the ultrasonic-based ETAs require the user to actively scan the environment to detect obstacles. The use of a robot as a navigation aid may encounter the mobility restriction problems (e.g., the stairs, tight spaces). Some ETAs suffer from the high-price problem.
Based on the aforementioned discussions, we propose an RGB-D camera-based travel aid to complement the use of a white cane. The proposed aid will greatly expand the environmental detection range of the blind person with a white cane. The prototype of the travel aid consists of a Kinect sensor, a portable computer, and an ear phone. The goal of the travel aid is to provide the user with the information for way finding. When a blind person is in an unfamiliar place, if the user can quickly get the information on the distances and orientations of obstacles in front of him or her then he or she can have more confidence in walking through the place via a white cane. The information about the obstacles will be output to the user via the earphone. The remainder of the paper is organized as follows. Section II presents the RGB-D camera-based travel aid for the blind. Section III describes the simulation results. Section IV gives the conclusions.

The RGB-D Camera-Based Travel Aid
The proposed RGB-D camera-based travel aid involves the following four stages.

Step 1: Image acquisition
The Kinect sensor acquires a depth image of the environment in front of the user.
The proposed travel aid greatly depends on the performance of the detection of ground plane. If we can successfully detect the ground plane from the image then the foreground (e.g., the obstacles, walls, etc.) can be correctly segmented. Recently, the RANSAC plane fitting [19] and the V-disparity image are two possible choices for detecting the ground plane. Each approach has its merits and limitations. In our approach, we fully utilize the information about the ground plane, ax by cz w + + = , provided by the Kinect SDK. The basic idea is the computation of the distance between a point, If the distance between a point and the ground plane is below a pre-specified threshold then this point can be declared to be a point belonging to the ground plane. Although this method is very straightforward, its performance is not high enough to be the only one criterion because the depth image is sometime noisy. Therefore, we use another criterion to further verify whether a point belongs to the ground plane. The basic idea is as follows. Let A denote the point, a b c , will be declared to be a point belonging to the ground plane. Figure 1 shows an example of the detection of the ground plane. Points do not belong to the detected ground plane are denoted as black points in Figure 1a. We still need to use two morphological operators (i.e., the erosion and dilation operators) to process Figure 1b to make the detected ground plane more smooth and consistent as shown in Figure 1c.

Step 3: Obstacle detection
In this step, we need to detect obstacles lying on the ground plane. The basic idea is as follows. We concern how far the user can reach. We use a morphological operator to fill out the holes inside the ground plane and then the locations of the holes will be the locations of the obstacles in front of the user. Figure 2 shows an example of the detection of obstacles.

Step 4: Audio signaling
The information about the detected objects (e.g., the distance between the obstacles and the user, the orientation of the obstacles, etc.) will be output to the user via the earphone.

Experiment one: The detection of the ground plane
In this experiment, we want to test the performance of our ground plane detection. Three different images with the ground truth information were collected as shown in Figure 3. Table 1 shows the performance of the proposed ground plane detection method described in step 2. It tells us that the proposed ground plane detection method can achieve high performance.

Criterion
Sensitivity Specificity Accuracy

Experiment two: The detection of the obstacles
In this experiment, we want to test whether the information (i.e., the height of the obstacles and the distance between the obstacle and the user) about the detected obstacles is correct. In this experiment, we have tested 10 different obstacles. Table 2 shows the results. It tells us that the information provided by the aid is useful since the average error ratios are 8.36% and 3.39%, respectively.

Conclusion
In this paper, we explore the possibility to applying an RGB-D camera in developing a vision-based travel aid for the blind or visually impaired people. A special ground plane detection method is developed. Based on the detected ground plane, we then can effectively detect obstacles in front of the user and provide the user with the information about the obstacles. We demonstrate that this idea, if efficiently and effectively implemented, can be really helpful to them.