Comment on Reliability and validity of the Chinese version of the Mild Behavioral Impairment Checklist for screening for Alzheimer’s disease

I read with interest the study of Cui et al. [1] assessing the reliability and validity of the Chinese version of Mild Behavioral Impairment-Checklist (MBI-C) in Alzheimer’s disease (AD) patients. The authors address key elements of the validation process, but their conclusions—the Chinese MBI-C is high in validity and reliability as well as superior to Neuropsychiatric Inventory Questionnaire—are unfortunately weakened by some persistent misconceptions of scale validation in general, and some common issues in neurodegenerative disease research. I raise these issues, as validity of neuropsychiatric symptom measures may have implications for clinical trials and diagnostics.

The authors present Cronbach’s alphas for MBI-C: 0.936 for the entire scale, and 0.878, 0.837, 0.863, 0.664, 0.824 for the five subscales, respectively [1]. For decades, psychometricians and methodologists have encouraged researchers to avoid relying on Cronbach’s alpha for scale validation, as the assumptions for using it are hardly ever met or even tested: namely, unidimensionality, tau equivalence, normal distribution of items and lack of covariance between item errors [2–4]. Alpha can also increase with the number of items, which means a “good” alpha may arise in scenarios with low inter-item correlations but excessive number of items in multidimensional data [2,3]. Internal consistency of the 34 MBI-C items that should theoretically represent five constructs, not one, could be one such scenario where an alpha > 0.9 was observed without it being psychometrically informative.

The authors acknowledge the necessity of conducting a confirmatory factor analysis in a larger sample to assess construct validity in future work, as their current sample size of 96 individuals (50 controls, 46 AD patients) is meager compared to common recommendations [5]. However, using principal components analysis, the authors extracted seven principal components, which appears to be interpreted as evidence for construct validity, even if it is in contrast with their examination of the original five subscale Cronbach’s alphas. To summarize, the construct validity evidence for the Chinese MBI-C remains inconclusive owing to a small sample size that renders confirmatory factor analysis inapplicable. Curiously enough, there was some data for reliability as well as sensitivity and specificity for AD diagnosis, which lends some merit to the authors’ notion that MBI-C could be used as an auxiliary part of the AD diagnostic tool kit in China without knowing whether it actually measures what it is supposed to. However, problems arise when phrases such as ‘MBI-C has high reliability and validity’ [1] are taken out of context.

The interplay between painstaking data collection and urgent demand for more sensitive methods is ever present in scale development and validation for neurodegenerative diseases. However, it should be borne in mind that only with large and representative samples should we presume to have validity evidence for a scale. It is beneficial for both researchers and patients that we allocate enough resources to conduct validation studies capable of providing convincing validity evidence for the scale in at least one setting. In general, accessible, non-technical guidelines for developing and validating scales (e.g., [5,6]) should be preferred over heuristics or adherence to the sometimes questionable practices set forth by previous measures.

Toni T. Saari
Institute of Clinical Medicine, Neurology, School of Medicine, University of Eastern Finland, Kuopio, Finland
School of Educational Sciences and Psychology, University of Eastern Finland, Joensuu, Finland

References
[1] Cui Y, Dai S, Miao Z, Zhong Y, Liu Y, Liu L, Jing D, Bai Y, Kong Y, Sun W, Li F, Guo Q, Rosa-Neto P, Gauthier S, Wu L (2019) Reliability and validity of the Chinese version of the Mild Behavioral Impairment Checklist for screening for Alzheimer’s disease. J Alzheimers Dis 70, 747–756.
[2] Cortina JM (1993) What is coefficient alpha? An examination of theory and applications. J Appl Psychol 78, 98–104.
[3] Tavakol M, Dennick R (2011) Making sense of Cronbach’s alpha. Int J Med Educ 2, 53–55.
[4] McNeish D (2017) Thanks coefficient alpha, we’ll take it from here. Psychol Methods 23, 1–23.
[5] Clark LA, Watson D (1995) Constructing validity basic issues in objective scale development. Psychol Assess 7, 309–319.
[6] Flake JK, Pek J, Hehman E (2017) Construct validation in social and personality research: current practice and recommendations. Soc Psychol Personal Sci 8, 370–378.