Data can become evidence only if it is reliable. Reliability is based on the degree to which the same observations can be made under similar circumstances but at different times, and also one the degree to which different measures of the same effect agree. Theory allows managers and leaders to make predictions about what they will observe. Reliable evidence is gathered from different sources; researchers seek find at least three sources of data to confirm reliable evidence.
Consider a situation in which IT managers have installed a learning management system (LMS) so teachers can supplement face-to-face activities with a virtual classroom, and they seek to ascertain the effect of the LMS on students’ grades. They would seek to answer the question, “Is use of the virtual classrooms associated with higher grades?” (This question assumes that grades are an accurate proxy for student learning and that learning is accurately reflected in course grades, as well as assuming teachers made no errors in creating or using the tests and other instruments used to generate student grades.)
By using three data sources, the IT managers can answer this question with reliable evidence. Here are three examples of data that would support the conclusion that the LMS was improving students’ grades. First, they can compare performance on a common test for groups that studied using the LMS and those that did not. For example, statistically significant differences in scores on a common Algebra exam between students in section that used the LMS compared to section that did not might indicate an effect of the LMS. Second, students cold be interviewed to ascertain their experience using the LMS; the qualitative data collected in this way will help explain differences observed (or not observed) in the exam scores described above. Third, the access logs could be analyzed to determine if there is a correction between the time a student spent on the LMS interacting with the materials and his or her scores on tests in the course. Of course the analysis of this last data would include steps to ensure the time on the LMS and not simply more time studying was associated with changes in grades.
As the LMS example illustrates, gathering reliable evidence necessitates careful attention to the conceptual artifacts under investigation and the nature of the data that will provide evidence. That example also demonstrates that both quantitative data that is analyzed through either descriptive or analytic methods and qualitative data that is analyzed through reading and coding.
Regardless of the type of data collected, researchers and efficacious IT managers must attend to sampling (how will subjects be selected) and they must ensure the instruments they are using are both valid (measuring what they claim to be measuring) and sufficiently precise for the purposes. In addition, IT managers have a responsibility to gather data in an ethical manner. For example, they must ensure the privacy of subjects, ensure the data do not pose a threat to them, and they must have an option to withdraw also with providing informed consent. These issues are particularly important for the populations of students who are minors, and for who informed consent must be explained to parents or guardians as well. In some cases, it is preferred to have data collected by individuals who are not teachers or in other positions of authority in school.