A smart wheelchair system using a combination of stereoscopic and spherical vision cameras

Nguyen, JS

A smart wheelchair system using a combination of stereoscopic and spherical vision cameras

Nguyen, JS

Permalink

Publication Type:: Thesis
Issue Date:: 2013

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download contents and abstractAdobe PDF (299.67 kB)

Adobe PDF

Download thesisAdobe PDF (11.49 MB)

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Nguyen, JS
dc.date.accessioned	2013-06-11T07:58:06Z
dc.date.available	2013-06-11T07:58:06Z
dc.date.issued	2013
dc.identifier.uri	http://hdl.handle.net/10453/21910
dc.description	University of Technology, Sydney. Faculty of Engineering and Information Technology.	en_US
dc.description.abstract	Reports have shown growing numbers of people who fall into the categories of the elderly or those living with some form of disability. Physical and functional impairments are broad-ranging across these groups and the causes are numerous, including strokes, spinal cord injury, spina bifida, multiple sclerosis, muscular dystrophy, and various degenerative disorders. Rehabilitation technologies are a solution to many of these impairments and aim to improve the quality of life for the people who require them. Smart wheelchair developments, in particular, have the purpose of assisting those with mobility disabilities. Providing independence in mobility can have many significant benefits to the users in their daily lives, including improved physical, cognitive, confidence, communication, and social skills. Unfortunately for many, particularly those with tetraplegia (partial or total loss of functionality through illness or injury to all four limbs and torso), there is a serious lack of options available for adequately and safely controlling mobility devices such as wheelchairs. There are few options for hands-free controlling wheelchairs, and furthermore, there are no accessible options for intelligent assistance from the wheelchair to make hands-free control easy and safe. This is a serious issue since the limited hands-free control options available can be difficult to use, resulting in many accidents. There are also new control technology devices emerging in research, such as brain-computer interfaces (BCIs), which could potentially provide an adequate means of control for many people who cannot use currently commercial options, but require intelligent assistance from the wheelchair to make use of such a system safe. In this thesis, the design and development of a new smart wheelchair, named TIM, is introduced to address these issues. The TIM smart wheelchair was created with the intention of providing intelligent assistance during navigation for any hands-free control technology, both currently commercial and new devices produced in research. This aims to vastly improve the options available to the people who are in need of such smart wheelchair developments. A method of utilising stereoscopic cameras for adaptive, real-time vision mapping is presented in this thesis, as cameras are increasingly becoming a more accessible and inexpensive form of artificial sensor. The mapping process in this method involves acquisition from the left and right stereo pair of cameras, which then undergo a range of image pre-processing techniques before being stereo-processed, which includes matching and correlation algorithms, to produce a disparity image. This disparity image contains depth information about the scene, which is then converted into a 3-dimensional (3D) point map, placing all mapped pixels of the environment, and features within, into a 3D plane. Unnecessary information, such as the floor and everything above the maximum height of the TIM smart wheelchair, is removed and the remaining data extracted into a 2-dimensional (2D) bird’s eye view environment map. This mapping representation assists the wheelchair in the later steps of making intelligent navigational decisions based on the relative placement of objects in the environment. Wheel encoders on the drive wheels are also acquired during operation, and odometry change calculations are performed to facilitate the ability of the system to ‘remember’ mapped object points that have passed outside the vision range. This is performed frequently to construct a real-time environment map, and to remember the placement of objects that have moved out of the range of vision, in order to further avoid collisions. This is particularly useful for static environments and creating maps of the static object placements within. A wheel parameter correction process was also employed to increase the accuracy of this mapping process and successfully reduce the errors associated with drive wheel specifications, which in turn can affect the mapping process based on the wheel encoder information. Correction of these parameters helped optimise the ‘memory mapping’ process and reduce skewing and accumulative errors in the maps. A process for intelligent stereo processing parameter selection was designed, as the quality of disparity images, and hence the quality of environmental-mapping, is heavily dependent on the stereo processing parameters, which may work well when set for one environment but produce problems in another. The differences that affected performance between environments were found to mostly be the lighting conditions which resulted from the varying types of environments. As such, this proposed method involves classifying environmental categories in real-time, based on image data available, and adapting the parameters accordingly. The environment types were separated into four categories to account for most encountered environmental situations, being 1) ‘General Illumination Contrast’, 2) ‘Extreme Illumination Contrast’, 3) ‘Consistent Dark’, and 4) ‘Consistent Bright’. The proposed method successfully allowed classification in real-time of the environment categories and adaptation of the stereo processing parameters in accordance, producing a system that can change its settings ‘on the fly’ to suit the environment the wheelchair is navigating through. Limited vision and trouble with dynamic objects were found to be downfalls with the stereoscopic vision, so to address these, methods of utilising a spherical vision camera system were introduced for obstacle detection over a wide vision range. Spherical vision is an extension of monoscopic cameras, producing 360º of panoramic vision. A strategy to utilise these panoramic images is presented, in which the images are separated into segments and ‘Traffic Light’ zones. The segments display different areas of the image representing the allocated areas around the wheelchair. The ‘Traffic Light’ zones within the segments are separated into three categories: 1) Red, meaning an obstruction is present around the wheelchair, 2) Yellow, indicating to take caution as an object is nearby, and 3) Green, meaning there are no objects close to the wheelchair in this segment. Image processing techniques have been assembled as a pre-processing strategy, and neural networks are used for intelligent classification of the segmented images into the zone categories. This method provides a wider range of vision than the stereoscopic cameras alone, and also takes into account the issue of detecting dynamic obstacles, such as people moving around. A unique combination of the stereoscopic cameras and the spherical vision cameras is then introduced. This combination and system configuration is biologically inspired by the equine vision system. Horses inherently have a large vision range, which includes a wide monocular vision range around and a binocular vision overlap ahead of the horse. In accordance with this effective vision system, the camera configuration on the TIM smart wheelchair was modelled similarly and advanced software integration strategies then followed. A method for advanced real-time obstacle avoidance is presented, which utilises algorithms in research, such as Vector Field Histogram (VFH) and Vector Polar Histogram (VPH) methods, and adapts them for use with the specified camera configuration. Further improvement upon the algorithms for this application provides safer obstacle avoidance during navigation in unknown environments, with an added emphasis on making automated navigational decisions towards areas with more available free space. Speed and manner of obstacle avoidance is dependent upon the placement spread of objects in an environment and how close they are to the wheelchair during navigation. Finally, the combination and integration of the automated guidance and obstacle avoidance capabilities of the wheelchair with hands-free control technologies are introduced. The aim of the TIM smart wheelchair system was to effectively provide safe navigation with automated obstacle avoidance in a manner that ultimately executes the user’s intentions for travel. As such, a head-movement control (HMC) device and a brain-computer interface (BCI) device are both separately integrated with the TIM smart wheelchair, providing a display of two new options for hands-free control. Experimental studies were conducted using these two control devices separately, to assess the performance of the TIM smart wheelchair, as well as its ability to carry out user’s navigational intentions safely and effectively. Eight able-bodied participants trialled the system, including four male and four female, with ages ranging from 21 to 56 years old. All of these able-bodied participants have not previously had any experience operating a wheelchair. In addition, two male tetraplegic (C-6 to C-7) participants also completed the experimental study, aged 20 and 33. Both tetraplegic participants are wheelchair users, so these experiments were of great importance. The same tasks applied to all, and included navigating obstacle courses with the use of the head-movement control system and the brain-computer interface for control. Experiment runs were conducted for each control system with automated navigational guidance assistance from TIM, and repeated for some capable participants without the assistance from TIM. This process was also conducted in two types of obstacle courses, being 1) a ‘Static Course’ and 2) a ‘Dynamic Course’, requiring different types of challenges in obstacle avoidance. This provided results to assess the performance and safety of the TIM smart wheelchair in a range of environments and situations. Evaluation of the results displayed the feasibility and effectiveness of the developed TIM smart wheelchair system. This system, once equipped with a unique camera configuration and reliable obstacle avoidance strategies, was able to successfully allow users to control the wheelchair with research-produced hands-free interface devices and effectively navigate safely through challenging environments. The TIM smart wheelchair system is able to adapt to people with various types and levels of physical impairment, and ultimately provide ease-of-use as well as safety during navigation.	en_US
dc.format	Thesis (PhD)	en_US
dc.language.iso	en	en_US
dc.relation	https://opus.lib.uts.edu.au/bitstream/10453/21910/5/02whole.pdf
dc.rights	au.edu.uts.lib/ppc
dc.rights	The author owns the copyright in this thesis including all reproduction and reuse rights for the work. The work may not be altered without the permission of the copyright owner. Attribution is essential when quoting or paraphrasing from this thesis.
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	Smart wheelchair.	en
dc.subject	Stereoscopic cameras.	en
dc.subject	Vision.	en
dc.title	A smart wheelchair system using a combination of stereoscopic and spherical vision cameras	en_US
dc.type	Thesis
utslib.copyright.status	open_access

Abstract:

Reports have shown growing numbers of people who fall into the categories of the elderly or those living with some form of disability. Physical and functional impairments are broad-ranging across these groups and the causes are numerous, including strokes, spinal cord injury, spina bifida, multiple sclerosis, muscular dystrophy, and various degenerative disorders. Rehabilitation technologies are a solution to many of these impairments and aim to improve the quality of life for the people who require them. Smart wheelchair developments, in particular, have the purpose of assisting those with mobility disabilities. Providing independence in mobility can have many significant benefits to the users in their daily lives, including improved physical, cognitive, confidence, communication, and social skills. Unfortunately for many, particularly those with tetraplegia (partial or total loss of functionality through illness or injury to all four limbs and torso), there is a serious lack of options available for adequately and safely controlling mobility devices such as wheelchairs. There are few options for hands-free controlling wheelchairs, and furthermore, there are no accessible options for intelligent assistance from the wheelchair to make hands-free control easy and safe. This is a serious issue since the limited hands-free control options available can be difficult to use, resulting in many accidents. There are also new control technology devices emerging in research, such as brain-computer interfaces (BCIs), which could potentially provide an adequate means of control for many people who cannot use currently commercial options, but require intelligent assistance from the wheelchair to make use of such a system safe. In this thesis, the design and development of a new smart wheelchair, named TIM, is introduced to address these issues. The TIM smart wheelchair was created with the intention of providing intelligent assistance during navigation for any hands-free control technology, both currently commercial and new devices produced in research. This aims to vastly improve the options available to the people who are in need of such smart wheelchair developments. A method of utilising stereoscopic cameras for adaptive, real-time vision mapping is presented in this thesis, as cameras are increasingly becoming a more accessible and inexpensive form of artificial sensor. The mapping process in this method involves acquisition from the left and right stereo pair of cameras, which then undergo a range of image pre-processing techniques before being stereo-processed, which includes matching and correlation algorithms, to produce a disparity image. This disparity image contains depth information about the scene, which is then converted into a 3-dimensional (3D) point map, placing all mapped pixels of the environment, and features within, into a 3D plane. Unnecessary information, such as the floor and everything above the maximum height of the TIM smart wheelchair, is removed and the remaining data extracted into a 2-dimensional (2D) bird’s eye view environment map. This mapping representation assists the wheelchair in the later steps of making intelligent navigational decisions based on the relative placement of objects in the environment. Wheel encoders on the drive wheels are also acquired during operation, and odometry change calculations are performed to facilitate the ability of the system to ‘remember’ mapped object points that have passed outside the vision range. This is performed frequently to construct a real-time environment map, and to remember the placement of objects that have moved out of the range of vision, in order to further avoid collisions. This is particularly useful for static environments and creating maps of the static object placements within. A wheel parameter correction process was also employed to increase the accuracy of this mapping process and successfully reduce the errors associated with drive wheel specifications, which in turn can affect the mapping process based on the wheel encoder information. Correction of these parameters helped optimise the ‘memory mapping’ process and reduce skewing and accumulative errors in the maps. A process for intelligent stereo processing parameter selection was designed, as the quality of disparity images, and hence the quality of environmental-mapping, is heavily dependent on the stereo processing parameters, which may work well when set for one environment but produce problems in another. The differences that affected performance between environments were found to mostly be the lighting conditions which resulted from the varying types of environments. As such, this proposed method involves classifying environmental categories in real-time, based on image data available, and adapting the parameters accordingly. The environment types were separated into four categories to account for most encountered environmental situations, being 1) ‘General Illumination Contrast’, 2) ‘Extreme Illumination Contrast’, 3) ‘Consistent Dark’, and 4) ‘Consistent Bright’. The proposed method successfully allowed classification in real-time of the environment categories and adaptation of the stereo processing parameters in accordance, producing a system that can change its settings ‘on the fly’ to suit the environment the wheelchair is navigating through. Limited vision and trouble with dynamic objects were found to be downfalls with the stereoscopic vision, so to address these, methods of utilising a spherical vision camera system were introduced for obstacle detection over a wide vision range. Spherical vision is an extension of monoscopic cameras, producing 360º of panoramic vision. A strategy to utilise these panoramic images is presented, in which the images are separated into segments and ‘Traffic Light’ zones. The segments display different areas of the image representing the allocated areas around the wheelchair. The ‘Traffic Light’ zones within the segments are separated into three categories: 1) Red, meaning an obstruction is present around the wheelchair, 2) Yellow, indicating to take caution as an object is nearby, and 3) Green, meaning there are no objects close to the wheelchair in this segment. Image processing techniques have been assembled as a pre-processing strategy, and neural networks are used for intelligent classification of the segmented images into the zone categories. This method provides a wider range of vision than the stereoscopic cameras alone, and also takes into account the issue of detecting dynamic obstacles, such as people moving around. A unique combination of the stereoscopic cameras and the spherical vision cameras is then introduced. This combination and system configuration is biologically inspired by the equine vision system. Horses inherently have a large vision range, which includes a wide monocular vision range around and a binocular vision overlap ahead of the horse. In accordance with this effective vision system, the camera configuration on the TIM smart wheelchair was modelled similarly and advanced software integration strategies then followed. A method for advanced real-time obstacle avoidance is presented, which utilises algorithms in research, such as Vector Field Histogram (VFH) and Vector Polar Histogram (VPH) methods, and adapts them for use with the specified camera configuration. Further improvement upon the algorithms for this application provides safer obstacle avoidance during navigation in unknown environments, with an added emphasis on making automated navigational decisions towards areas with more available free space. Speed and manner of obstacle avoidance is dependent upon the placement spread of objects in an environment and how close they are to the wheelchair during navigation. Finally, the combination and integration of the automated guidance and obstacle avoidance capabilities of the wheelchair with hands-free control technologies are introduced. The aim of the TIM smart wheelchair system was to effectively provide safe navigation with automated obstacle avoidance in a manner that ultimately executes the user’s intentions for travel. As such, a head-movement control (HMC) device and a brain-computer interface (BCI) device are both separately integrated with the TIM smart wheelchair, providing a display of two new options for hands-free control. Experimental studies were conducted using these two control devices separately, to assess the performance of the TIM smart wheelchair, as well as its ability to carry out user’s navigational intentions safely and effectively. Eight able-bodied participants trialled the system, including four male and four female, with ages ranging from 21 to 56 years old. All of these able-bodied participants have not previously had any experience operating a wheelchair. In addition, two male tetraplegic (C-6 to C-7) participants also completed the experimental study, aged 20 and 33. Both tetraplegic participants are wheelchair users, so these experiments were of great importance. The same tasks applied to all, and included navigating obstacle courses with the use of the head-movement control system and the brain-computer interface for control. Experiment runs were conducted for each control system with automated navigational guidance assistance from TIM, and repeated for some capable participants without the assistance from TIM. This process was also conducted in two types of obstacle courses, being 1) a ‘Static Course’ and 2) a ‘Dynamic Course’, requiring different types of challenges in obstacle avoidance. This provided results to assess the performance and safety of the TIM smart wheelchair in a range of environments and situations. Evaluation of the results displayed the feasibility and effectiveness of the developed TIM smart wheelchair system. This system, once equipped with a unique camera configuration and reliable obstacle avoidance strategies, was able to successfully allow users to control the wheelchair with research-produced hands-free interface devices and effectively navigate safely through challenging environments. The TIM smart wheelchair system is able to adapt to people with various types and levels of physical impairment, and ultimately provide ease-of-use as well as safety during navigation.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/21910