Cleansing Noisy Data Streams

IEEE Computer Society
Publication Type:
Conference Proceeding
2008 Eighth IEEE International Conference on Data Mining, 2008, pp. 1139 - 1144
Issue Date:
Full metadata record
Files in This Item:
Filename Description SizeFormat
2008001693OK.pdf1 MBAdobe PDF
We identify a new research problem on cleansing noisy data streams which contain incorrectly labeled training examples. The objective is to accurately identify and remove mislabeled data, such that the prediction models built from the cleansed streams can be more accurate than the ones trained from the raw noisy streams. For this purpose, we first use bias-variance decomposition to derive a maximum variance margin (MVM) principle for stream data cleansing. Following this principle, we further propose a local and global filtering (LgF) framework to combine the strength of local noise filtering (within one single data chunk) and global noise filtering (across a number of adjacent data chunks) to identify erroneous data. Experimental results on six data streams (including two real-world data streams) demonstrate that LgF significantly outperforms simple methods in identifying noisy examples.
Please use this identifier to cite or link to this item: