I posted on social media recently:
Can educators devise assessments that can predict how students will develop rather than what they did?
I was pushed to explain myself and that got me thinking what exactly I meant. So, here goes!
We all know the culture of assessment in education. We are charged with documenting what we do and gathering data to support our decisions and to document student performance. As an educator who was trained prior to the emergence of this culture and whose career before the hyperfocus on data lasted more than a decade, it is my opinion that we did not gain as much from our focus on data as some advocates claim.
Please do not misunderstand my position. I do believe we can and should assess students’ learning. Thinking about what we want them to learn and how they will demonstrate it does improve our courses. What does not help improve courses is oversharing too specific learning outcomes with students (it just introduces extraneous cognitive load as students have to track more information about the lesson which leaves less capacity to learn the lesson). What else does not help our students is a formal measurement for every step of the way.
The preceding paragraph can be summarized as: “I trust students to pay attention to well-structured lessons focused on relevant and important questions. I trust teachers to observe students and make judgements of how they’re doing. But formal measurements can help.” (I am not pollyannish in this view; I know students will game the system and teachers will be biased. I do think classrooms that do not trust students to learn and teachers to teach are less educative than those that do.)
Ove my career, I have also thought about testing (which continues to be the dominant source of data in education settings), and the different meanings and purposes of “test” can confuse folks who are talking about them. We know:
- Self-testing, in which students reconstruct what they learned previously, improves learning.
- Teachers construct and administer tests that comprise questions that are supposed to measure student’s learning. These are localized (focusing on the content taught by the teachers to that particular group) and rarely evaluated (we can but usually) find out switch questions were missed, why they were missed, or which questions help us discriminate performance.
- Standards-based tests are supposed to measure the degree to which students know the materials defined in the curriculum.
- Standardized tests measure an individual’s performance on a test compared to other test takers.
- Some tests are designed to be culminating activities. We have all felt the relief that accompanies passing an exam in a course that we intend to be our final one in the subject.
- Some tests are designed to be gateway activities. Professional licensing exams are perhaps the most familiar example.
As an educator, I am slightly interested in those tests that measure the degree to which my students accomplished the goals that I set for them. Because they happen after the teaching, these tests or assessments have limited usefulness for the future actions of the students. They may help me to improve next year’s lessons (which will be delivered to a new group of students) and it may point out deficiencies the next teachers will face or recommend remediation, but they look back.
I think most who care about student performance are more interested in students’ futures. Some tests do have this predictive value. We can reasonably predict that students who engage in self-testing will perform better on future tests and assessments. The predictive value comes from the correlation of the practice and future scores and the casual explanation; we can claim “if you self-test, you will learn the material better because you are strengthening the neural connections to use the information later.” We can also make some predictions using large data sets and professional exams. I trust the professions nurses to refine their tests so that scores correlate with improved performance in clinical settings.
I often envy the predictive value of such tests, but they were developed in conditions unlike the tests created and used in most educational settings.
- We do not have a community of scholars investigating our classroom tests that have elucidated the effectiveness of self-testing. Research proceeds on much longer timelines than we have in classrooms. We must do the best we can i the time we have to give students (and ourselves) actionable information.
- We do not have a trackable cohort of students with known data points like they have in nursing. We can track those who stay licensed and work in the field and we have reports on how well they perform in the field. I trust nurses are using this data to constantly improve the predictive value of their professional exams. We lose track of our students, and we cannot compel their continued participation in our data collection.
- Those who work in general education are also preparing students for unpredictable futures. If we do not know what path students plan to take, we cannot know the tests or assessments that will correlate with success and the causative explanations would be dubious at best.
As I get to the end of this blog post, I am rethinking my social media post. I am not sure we want a test that predicts success. If attempted to create predictive tests for general education purposes, they would be biased and probably correlate more closely with the socioeconomic status of the student than they success. We value privacy, so students should be able to go about their lives without reporting back to us how they are doing. It takes a special type of hubris to think we know what test would predict what students will need in the future. This is one reason professional exams are under constant revision and are obsolete when they are released.
Maybe I can be content if educators simply be realistic about the tests and measures they use. Realize they are not as valid and reliable and objective as we think (or publishers claim). Realize they always give information too late. Realize they can always be improved.
Realize they are not a substitute for connecting with students.