Innovations in Competence Assessment: Design and Initial Validation of the Vignette Matching Assessment Tool (VMAT)

Gonsalvez, CJ; Deane, FP; Terry, J; Nasstasia, Y; Shires, A

Innovations in Competence Assessment: Design and Initial Validation of the Vignette Matching Assessment Tool (VMAT)

Gonsalvez, CJ Deane, FP Terry, J Nasstasia, Y Shires, A

Permalink

Publisher:: American Psychological Association
Publication Type:: Journal Article
Citation:: Training and Education in Professional Psychology, 2021, 15, (2), pp. 106-116
Issue Date:: 2021-01-01

Closed Access

	Filename	Description	Size
	ContentServer.pdf	Published Version	471.86 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Gonsalvez, CJ
dc.contributor.author	Deane, FP
dc.contributor.author	Terry, J
dc.contributor.author	Nasstasia, Y
dc.contributor.author	Shires, A https://orcid.org/0000-0001-8153-1126
dc.date.accessioned	2021-11-10T02:49:55Z
dc.date.available	2021-11-10T02:49:55Z
dc.date.issued	2021-01-01
dc.identifier.citation	Training and Education in Professional Psychology, 2021, 15, (2), pp. 106-116
dc.identifier.issn	1931-3918
dc.identifier.issn	1931-3926
dc.identifier.uri	http://hdl.handle.net/10453/151476
dc.description.abstract	This study responds to an urgent need for innovative approaches to competence assessment in the face of strong evidence that several current instruments may be vulnerable to systematic rating biases. Building on previous research, the current study aimed to (a) design a full catalogue of vignettes for the Vignette Matching Assessment Tool (VMAT); (b) establish a matrix of calibration scores for the catalogue of vignettes by having experts rate each of the vignettes; (c) evaluate its psychometric properties by comparing competence ratings derived from the VMAT with those from a conventional instrument, the Clinical Psychology Practicum Competencies Rating Scale (CΨPRS); and (d) assess whether the VMAT reduced halo and leniency biases. An initial catalogue of 52 vignettes was recalibrated by a large sample of experts (N = 45) to establish a final catalogue of 41 standardized vignettes across 10 domains and multiple stages of competence. The VMAT used the matrix of calibrated vignettes to help supervisors anchor competence trajectories attained by trainees for each of 10 competency domains. The results provide initial support for the validation of the VMAT. The VMAT attenuated halo effects, and most supervisors endorsed the VMAT as their preferred instrument. Compared to the C±PRS, cutoff scores on the VMAT identified a larger number of trainees with serious competence problems (n = 3 vs. n = 1). The current study has important implications for competence assessments and benchmarking within psychology and valuable cross-disciplinary applications.
dc.language	English
dc.publisher	American Psychological Association
dc.relation.ispartof	Training and Education in Professional Psychology
dc.relation.isbasedon	10.1037/tep0000302
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject	1302 Curriculum and Pedagogy, 1303 Specialist Studies in Education, 1701 Psychology
dc.subject.classification	Clinical Psychology
dc.title	Innovations in Competence Assessment: Design and Initial Validation of the Vignette Matching Assessment Tool (VMAT)
dc.type	Journal Article
utslib.citation.volume	15
utslib.for	1302 Curriculum and Pedagogy
utslib.for	1303 Specialist Studies in Education
utslib.for	1701 Psychology
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Health
pubs.organisational-group	/University of Technology Sydney/Faculty of Health/Graduate School of Health
pubs.organisational-group	/University of Technology Sydney/Faculty of Health/Graduate School of Health/GSH.Clinical Psychology
utslib.copyright.status	closed_access	*
pubs.consider-herdc	true
dc.date.updated	2021-11-10T02:49:53Z
pubs.issue	2
pubs.publication-status	Published
pubs.volume	15
utslib.citation.issue	2

Abstract:

This study responds to an urgent need for innovative approaches to competence assessment in the face of strong evidence that several current instruments may be vulnerable to systematic rating biases. Building on previous research, the current study aimed to (a) design a full catalogue of vignettes for the Vignette Matching Assessment Tool (VMAT); (b) establish a matrix of calibration scores for the catalogue of vignettes by having experts rate each of the vignettes; (c) evaluate its psychometric properties by comparing competence ratings derived from the VMAT with those from a conventional instrument, the Clinical Psychology Practicum Competencies Rating Scale (CΨPRS); and (d) assess whether the VMAT reduced halo and leniency biases. An initial catalogue of 52 vignettes was recalibrated by a large sample of experts (N = 45) to establish a final catalogue of 41 standardized vignettes across 10 domains and multiple stages of competence. The VMAT used the matrix of calibrated vignettes to help supervisors anchor competence trajectories attained by trainees for each of 10 competency domains. The results provide initial support for the validation of the VMAT. The VMAT attenuated halo effects, and most supervisors endorsed the VMAT as their preferred instrument. Compared to the C±PRS, cutoff scores on the VMAT identified a larger number of trainees with serious competence problems (n = 3 vs. n = 1). The current study has important implications for competence assessments and benchmarking within psychology and valuable cross-disciplinary applications.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/151476