Fusion of optical images with radar images constructed using pulsed doppler beamforming to enhance object tracking : creating a novel five layer image augmented with radial velocity and distance properties
- Publication Type:
- Thesis
- Issue Date:
- 2015
Open Access
Copyright Clearance Process
- Recently Added
- In Progress
- Open Access
This item is open access.
Target tracking that detects the presence, color, contour, position, velocity, and acceleration is very important in intelligent systems. Single or multiple sensor fusion methods have been developed to improve the tracking quality. There are algorithms to fuse radar and optical images at a decision-making level. However, not many algorithms have been applied to radar image and optical image fusion. This thesis investigates the fusion of these two images. A new five-layer image format is proposed.
Beamforming, a radar signal processing method, has been applied to remote sensing. This suggests the idea that beamforming can be used to create images similar to optical images, with additional information such as range and radial velocity. The radar image can be fused with the optical image to enhance target tracking.
This thesis attempts to prove this proposition. First a tracking system with an optical imaging subsystem and a radar imaging subsystem was designed. Optical images were captured by a high-quality webcam, and radar images were captured by a system designed and built by the author. The radar subsystem is a short-range airborne acoustic radar which contains the transmitter and receiver board, FPGA board and a PC. All the channels need to work simultaneously. The data rate and computation load were very high when the number of sensors increased. For a low-cost development, the resources of FPGA are very limited. Hardware construction is central to this thesis.
Next was the development of a mathematical model of the range layer using pulsed beamforming. The performance with multiple targets was evaluated. The relationship between angle resolution and sensor number was investigated. Linear sensor array beamforming was extended to two dimensions to create the image. It was shown that the targets could be clearly extracted. The required resources increased linearly as resolution improved while on the contrary for a full rectangular array the required resources increased exponentially. Improved image resolution with fewer resources was investigated. Structures included two-line multiplication beamforming, crossed array beamforming, and other sparse structures. Their performances were tested with added noise. Results showed that the algorithms were robust. Sparse rectangular sensor arrays may have ambiguity. The pulsed mode was adopted to detect the distance. A matched filter was used to maximize SNR and minimize false alarms.
In addition to pulsed beamforming, Doppler Beamforming was adopted to provide radial velocity information, which is important for detecting moving objects. This can remove stationary "clutter" from the image. The performance of linear sensor array beamforming was analyzed for both single and multiple targets. Sensor spacing and angle of arrival were investigated. Then the beamforming was extended to two dimensions. Full rectangular array beamforming, two-line multiplication beamforming, crossed sensor array beamforming and sparse rectangular array beamforming and were studied. FFT filtering methods and filter bank methods were investigated. The processing algorithm and the example image were provided.
Five fusion scenarios for the proposed new image format are discussed.
• Scenario 1 is the fusion of video camera frames with Doppler beamforming radar to enhance moving object detection. The fusion of frame differencing with Doppler beamforming was studied.
• Scenario 2 is using imaging to reduce the ambiguity of crossed array beamforming. The template matching and frame differencing were used to process the region which may contain targets. Then a decision making strategy for reducing ambiguity was introduced.
• Scenario 3 is the fusion of optical imaging with Doppler beamforming radar to enhance tennis ball tracking. The region of interest was marked by the radar. The optical imaging was used to judge the in/out of the tennis ball.
• Scenario 4 is the fusion of optical imaging with pulsed beamforming radar to track a target in 3D position. The Kalman filter was studied and two fusion algorithms were researched.
• Scenario 5 is using Doppler beamforming radar to improve gait analysis. The micro-Doppler characteristics are used to analyze human movement. Then Doppler processing was combined with Doppler beamforming to relate the Doppler frequency to the body part, which can be projected into the optical image.
The experiments show that the creation of a range image layer and a radial velocity image layer are successful. The proposed two extra image layers can be combined with a traditional RGB image to enhance object tracking. The disadvantage is that the data rate is very high when the sensor number is increased. In two dimensional beamforming, there is ambiguity in some shapes of sensor arrays. A sparse rectangular array can be used to reduce the sensor number with fewer ambiguities. The ambiguities can be removed by fusion of different layers in the new image format. The new image format and the fusion algorithms can bring great value to the area of surveillance, sports, medical care and computer-human interaction.
Please use this identifier to cite or link to this item: