Front-liner #2: Be Careful With “Data.”
One of the words you hear early and often when you get into the education reform game is ‘data.’ We all know there are big problems in our highest-need schools– systemic problems that leave many of our students years behind in basic thinking skills. When I taught with Teach For America in Tulsa Public Schools back in 2009-2012, I believe the data suggested that less than 7% of TPS high school graduates were “4-year-college-ready.” This was an incredibly powerful data point for me. I never forgot learning it and thought of it at least once per week for all three of my years in TPS. Mind you, these were the GRADUATES. This data did not account for the large numbers of students who never made it to graduation. The cold, hard fact revealed by the study that had produced this data point left no room for sugarcoating or delusion.
Data can be an important spur for change. But we need to be careful with the conclusions we draw from it. In this “post-truth” society, people too often see what they already want to believe in data, impose an already-set worldview on it, and draw their “action” conclusions accordingly. Alternatively, they can implement and persist in completely ineffective solutions because they are seemingly backed by “the numbers.” This is a really huge subject, but I’d like to discuss an example of each of these problems drawn from my experience as a TFA corps member and KIPP teacher.
In the former case, I often think of the data (quoted by many friends of mine) that shows quite conclusively on a nation-wide basis that black students lag behind white students in reading comprehension and math skills. Given our nation’s troubled racial history, many people outside of the education field ‘put two and two together’ and conclude “THE SYSTEM IS RACIST.”
Zoom in on Tulsa in particular, compare the comprehension levels of my former middle school (95 percent black population) with one of the suburban middle schools (90 percent white population), and you’ll see a similar gap.
Racism? It’s a good question.
But while it would be really great to narrow the problems for my former students down to one social evil we could attack and destroy, and while I can’t claim that the vast American education system is free from racism, my individual experience taught me that this would have been a wrong-headed conclusion to draw at best. My school had several black teachers; there were two on my grade-level team of four “core class” instructors. My students measurably learned from their black social studies teacher and measurably did NOT learn from their black math teacher. My school’s Dean of Students, Vice Principal, and security officer were all black and grew up locally. This had no effect on the many problems my school had with its learning environment: problems that led to its closure after my second year.
And then there were my 4-5 percent white students. For as much as the word “white privilege” gets thrown around by completely ignorant people, you’d think those students must have had it made. Aside from being the frequent target of racially-motivated and violent bullying, those students were by and large just as behind (if not more so) than their peers. I keep in touch with many of my former students from all 8 of my previous teaching years, and can say for a disturbing fact that not a single one of my white students from my first year of teaching graduated high school. This system supposedly rigged in their favor failed them entirely. My kids of all skin colors faced problems in broken or criminalized families, gang-infested neighborhoods, constantly shifting living circumstances, indifferent teachers and administrators of all types. Abuse, neglect, and tragedy were the “normal” for their lives. Irrespective of data showing that white students at a suburban school were far ahead of black students at my school, “racism” is foolish and fantastical scapegoating.
Then there are the conclusions that lead to ineffective action and persist because THE DATA SAYS THIS WORKS. Case in point: my original KIPP school leader was a hard fighter for student achievement and built a culture that (while problematic in many respects) drove that achievement in a measurable way for our kids. In tested thinking skills, they were outperforming their peers from the same neighborhoods in public schools by a wide margin. But as far as action drawn from data-backed conclusions, we were throwing literally everything at the wall to see what stuck, and sticking with some things for far too long.
Take these three data points: 1. Students generally fare better in charter schools with longer school days. 2. Data suggests that eating breakfast every morning leads to better school performance. 3. Dedicated reading time of 20-30 minutes per day raises reading comprehension. All three of these conclusions seem solid enough, although the science behind number 2 has been questioned recently for (whattaya know?) being used to support the wrong conclusion. Leave that aside for a moment, though, and let’s just assume there is good enough reason to believe all three of these data-backed conclusions.
In the four years prior to my arrival at the school and for the first three years of my tenure there, teaching staff had a massive school day by regular standards: required arrival by 6:55 AM and departure no earlier than 5 PM. Teachers were “on-duty” supervising lunch (in a cacophonous and chaotic lunch room) and limited to a single “plan” period of about 70 minutes. At least 8 hours and 45 minutes per day was spent imposing a fairly rigid interpretation of KIPP culture on rooms full of students, including passing periods, in which all students (5-8th grade) were required to walk in straight, silent lines from class to class. As you can imagine, these duties imposed a tremendous psychological strain on the classroom teacher.
Somewhat ironically, one of the greatest causes of strain for the least amount of reward was a period that attempted to combine all three of those previously-listed data points. Students arrived by bus from 7:05-7:15, were required as per our qualification for federally-funded breakfast to ALL proceed to the cafeteria and pick up free breakfast trays, then travel to their “homeroom” classrooms to eat breakfast and (ideally) read books until first period began at 8. Everyone eats breakfast, everyone has a longer school day, and everyone reads 20-30 minutes. Awesome, right?
As it turned out empirically, not at all. The vast majority of students either did not like the breakfast food or would skip eating entirely. Teachers watched that (taxpayer-funded) food go straight into the trash can. Meanwhile, duty insisted that silence reined in the classroom for reading time, during which many of our 7th and 8th grade kids, resentful of a school day that had them on the bus by 6:20 in the morning and finally off the bus past 5PM at night, barely pretended to read. During this time, our teachers who already struggled with classroom management and student behavior problems, would be put through the ringer. On top of this, teachers themselves were often poor models of the reading habits we were supposed to encourage the students to embrace. “Do as I say, not as I do”– hardly a recipe for success with 12-14 year olds.
The result: complete failure in all three objectives for most of our staff, combined with a living nightmare in behavior management. By 8AM, instruction had not even begun and our rookie teachers were ready for the day to be over. Only nine more hours to go! There were teachers who facilitated genuine reading time for some students and a small handful of kids who needed the breakfast and got it, but by and large, “advisory time” as this opening time period was called, was an abject failure by every tangible and intangible measure. Even worse, far from being an addition to school culture, it actively destroyed school culture. This remained true from before my first day in the fall of 2012 until we finally made real adjustments in the spring of 2016.
Finally, after years of negative feedback and teacher burnout, my local KIPP network looked at ways to shorten the school day. It wasn’t long before “advisory” was targeted (along with some other additional non-instructional teacher duties) and we chopped a full half hour off of the school day and almost an hour off of teacher duty time. Our new school leader and the executives of my local network did an awesome job on combined problem-solving. But anybody actually in classrooms and observing from a detached perspective could have concluded after the first two months of the original “advisory” plan that it was a fruitless and pointless disaster. How could we have stuck with it for so long? The question was asked many times from 2012-2015 and the answer always revolved around the dogged belief in THE DATA SAYS THIS WORKS. Once again– all context removed from the equation, well-meaning and high-minded people gather around the wrong tree and bark all day long.
Data, especially “big” data on a large scale, has an important place in diagnosing our problems. But in our pursuit of “rock-solid,” impersonal math and science to prove our conclusions beyond all emotional human error, I’ve seen (far too many times) how we can make our greatest errors and (far more egregiously) fail to learn from them.