Multi-source item content, face validity and ecological-digital validity: Pretesting with experts, novices, and ChatGPT-4o in the development of the preliminary Observing Parent-Child Interaction Inventory (OPCII).

Müller, J.M.

Abstract

Background: Many observational instruments have been developed to measure aspects of the parent-child relationship or interaction patterns. However, given the flood of terms and scale labels, these instruments often suffer from a lack of factorial, content, convergent or discriminant validity. Research Question: We examine how one can assess content validity during item pretesting and test development using multi-source information from experts, novices and ChatGPT-4o. We ask: Are ratings on the suitability of items consistent within and between these groups? Does content validity depend on scale characteristics such as construct popularity or category breadth? Method: The newly developed parent-child interaction inventory, namely the Observing Parent-Child Interaction Inventory (OPCII-1.0), comprises 19 scales with a total of 460 items. Each item was rated independently by six experts, six novices and six ChatGPT-4o prompts. A linear mixed model was applied to analyze the influence of group membership and item pool characteristics, with repeated measures nested within raters. Results: Mean score differences emerged across groups, with experts rating items most conservatively, ChatGPT-4o most liberally and novices falling in between. Additionally, item pools differed significantly in terms of their average suitability scores. An exploratory factor analysis of rater agreement revealed that ChatGPT-4o ratings showed the highest and most consistent loading on a common factor of item suitability. Discussion: Our multi-source evaluation provides evidence for content, face and ecological-digital validity. By implementing a transparent methodology—including detailed item generation instructions—we enhance the replicability of content validity assessments. This approach aims to initiate a convergent development process following decades of divergent construction of parent-child interaction instruments.