Harnessing Confidence for Report Aggregation in Crowdsourcing Environments
- Publisher:
- IEEE
- Publication Type:
- Conference Proceeding
- Citation:
- 2022 IEEE International Conference on Services Computing (SCC), 2022, 00, pp. 305-314
- Issue Date:
- 2022-01-01
Recently Added
| Filename | Description | Size | |||
|---|---|---|---|---|---|
| Harnessing Confidence for Report Aggregation in Crowdsourcing Environments.pdf | Accepted version | 593.18 kB |
Copyright Clearance Process
- Recently Added
- In Progress
- Open Access
This item is new to OPUS and is not currently available.
Crowdsourcing is an effective means of accomplishing human intelligence tasks by leveraging the collective wisdom of crowds. Given reports of various accuracy degrees from workers, it is important to make wise use of these reports to derive accurate task results. Intuitively, a task result derived from a sufficient number of reports bears lower uncertainty, and higher uncertainty otherwise. Existing report aggregation research, however, has largely neglected the above uncertainty issue. In this regard, we propose a novel report aggregation framework that defines and incorporates a new confidence measure to quantify the uncertainty associated with tasks and workers, thereby enhancing result accuracy. In particular, we employ a link analysis approach to propagate confidence information, subgraph extraction techniques to prioritize workers, and a progressive approach to gradually explore and consolidate workers’ reports associated with less confident workers and tasks. The framework is generic enough to be combined with existing report aggregation methods. Experiments on four real-world datasets show it improves the accuracy of several competitive state-of-the-art methods.
Please use this identifier to cite or link to this item: