“Data” has been widely, but imprecisely, used in education for most of the 21st century. Data-driven educators make decisions based on information they have gathered about their students’ performance. Ostensibly, this is done in an attempt to adopt the position of a researcher and to ground decisions in objective research, thus give more support for their decisions. Upon closer inspection, however, there is little resemblance between data collection, the data, and the analysis methods used by researchers and those used by most “data-driven educators.”
Data-driven educators tend to use data that is conveniently available; this data is almost always scores on a standardized or standards-based tests. These tests include both large scale and high stakes tests and also those administered by teachers in the classroom for diagnostic purposes. The validity and reliability of these tests is rarely questioned; educators who claim to be “data-driven” accept that the tests accurately measure what the publishers claim. Data-driven educators also tend to seek interesting and telling trends in the data, but rarely do they seek to answer specific questions using their data. Further, they rarely use theory to interpret results; it is assumed that instruction determined the scores and that changes to instruction affected all trends they observe.
Researchers, on the other hand, define the questions they seek to answer and the data methods they will use prior to gathering data; they gather only the data they need, and all data is interpreted in light of theory. Researchers challenge themselves and their peers to justify all assumptions and to demonstrate the validity and reliability of instruments that generate data and they challenge themselves and peers to demonstrate the quality of their data and conclusions; for researchers, conclusions based on invalid or badly (or unethically) collected data must be discarded by credible researchers and managers.
By adopting a stance towards data that more closely resembles research than data-driven decision-making, IT managers tend to base their decisions in data that is more valid and reliable than is commonly used in education. Their decisions are also more likely to be grounded in theory that helps explain the observations. Other benefits of adopting a research-like stance towards data and evidence include:
- More efficient processes as planners use theory to focus efforts on relevant factors and only relevant factors;
- More effective decisions, because multiple reliable and valid data are used;
- More effective interventions, because they focus on locally important factors and there is a clear rationale for actions;
- Assessments and evaluations of interventions are more accurate and more informative for further efforts because evidence is clear and clearly understood.
Research is generally differentiated into two types. Pure research is designed to generate and test theory, which contains ideas about how phenomena work and allows researchers to predict and explain what they observe. Applied research is undertaken to develop useful technologies that leverage the discoveries of pure research; applied research is often called technology development. Scholars who engage in pure research identify and provide evidence for cause and effect relationships; this is typically done through tightly controlled experiments and quantitative data. Scholars and practitioners who engage in applied research or technology development seek to produce efficient and effective tools (see figure 7.1).
