For the last 20 years or more, we have expected every leader to be “data-driven.” If they don’t claim to ground decisions in data, then it is assumed they are just making up what they want to do. Despite the chatter about data, I have found those who make the strongest claims to be data-driven have the least well-developed concepts of data and are among the least critical of data.
One signal that I use to identify those who are the least critical users of data is their inability to speak clearly about their data. Consider a test. Ostensibly, it measures each student’s abilities to perform certain tasks. I will assume the test is valid (what is being measured is a real “thing” and the test actually measures it) and reliable (the test returns similar results for those with similar skills). We could challenge those assumptions for many tests, but that is for a different post.
Responsible and critical users of data will realize the test collection of test scores comprise:
- Accurate data—These are the scores that correctly reflect what students know.
- Error—These are the scores that do to reflect what students know. This will include questions that students purposefully answered wrong, the questions that were inaccurately written, the misunderstood questions, and other inaccuracies due to design.
- Noise—These are the data resulting from random situations. A student may mismark their answer. They may forget to answer a question. The teacher may mismark the answers.
Data–driven leaders who are uncritical accept all data as accurate. That may make sense given the fact that if one considers every potential interpretation of data, then one may be unable to ever make a decision. I am more confident in the data skills and decisions of leaders who are at least aware that their data contain error and noise.