All measurements are subject to error; this is well-known to carpenters who “measure twice and cut once,” and it is known to observers of elections in which recounts result in much different returns than the original count. Measurements in schools are subject to error as well. If a student is handed back a quiz with nine questions out of 10 marked correct, the student (her parents) and the teacher all assume that she understands those questions and knows the correct answer, and presumably knows more than the student who scored only eight out of 10 on the same quiz. Some reflection suggests how those measures could be in error:
- the student may have simply guessed the correct answer, and may not know it;
- the student may have misunderstood the question, and thus been answering a different question that happened to have the same answer as the question being asked;
- the student may have incorrectly answered a misunderstood question, which happened to be the correct answer to the asked question;
- the teacher may have made errors in deciding which was the correct answer;
- the teacher may have marked an incorrect answer as correct.
Such errors happen all of the time in educational measurement, but those are rarely recognized, and pointing out those potential errors affect the grades teachers report is met with indignation. (That last statement comes from my experience.)
To minimize the damage done to data and conclusions by similar errors in their measurements, scientists draw conclusions based on large numbers of measurements (statisticians use “N” to indicate the number of measurements used in the statistical analysis). In general, the larger the N, the more reliable the measurements, and observations based on small N are viewed with skepticism.
Large N allows scientists to use statistics to support conclusions, with means, standard deviations, variance and similar numbers. When reporting an individual’ results, scientists cannot use the statistics of large N, so a result is reported as a range. Given this, all individual results are in reality approximations.
Scientists are also aware of variation of measurement. In the 21st century, it has become common for a the test results of the students in various schools to be reported in the news. Leaders of those schools that score near the top of the range, hold that up as evidence that they are doing well, and those at the bottom of the range promise to redouble efforts. Because of an effect know as regression to the mean, those who are long in the range are likely to see scores closer to the mean in the next year.
Further, science is open and transparent about the measurements and the instruments used to measure. Those instruments are subject to validation and reliability studies in the same way that all other aspects of the scientific endeavor are.