In this article, Professor Martin West discusses the limitations of measuring non-cognitive skills, particularly the reliability of self-report measures typically used to assesses these skills. While self-report questionnaires have advantages in being quick, inexpensive, and easy to administer, they are subject to bias, namely social desirability and reference bias. West describes a research study he conducted to illustrate the biases found in self-reported measures of non-cognitive skills. Based on the results of the study, West cautions the use of current survey-based measures of non-cognitive skills in high-stakes accountability systems and calls for researchers and educators seeking to enhance students’ non-cognitive skills to develop alternative measures that are valid across a broad range of school settings.
West is most concerned with reference bias, "which occurs when survey responses are influenced by differing standards of comparison." For example, when a student is asked to decide whether she is a hard worker, she needs to conjure up a mental image of hard work and compare her own habits to this mental image. That mental image is different for students depending on their context (e.g., school culture, family influences). This is where self-report measures are unreliable and difficult to compare results across different school settings. He uses his own research to show how middle schools students who were administered self-report surveys for non-cognitive skills showed reference bias based on the type of school they attended. West concludes that "it appears that existing survey-based measures of non-cognitive skills, although perhaps useful for making comparisons among students within the same educational environment, are inadequate to gauge the effectiveness of schools, teachers, or interventions in cultivating the development of those skills. Evaluations of the effects of teacher, school, and family influences on the development of non-cognitive skills could lead to false conclusions if the assessments used are biased by distinct frames of reference." This article provides important data and evidence to illustrate concerns that researchers have about using current measures related to social and emotional learning in accountability systems.