Coaching Letter #212
On conditioning on the wrong variable
Hello, I hope this finds you well. Thank you for subscribing to the Coaching Letter. You rock. In particular, thank you to all the very many people who responded to the last few Coaching Letters on equity. I can’t tell you how much I appreciate it.
It’s hard to believe it’s March already, and that it’s five years since the start of the pandemic lockdown. So much has happened since then, and I am very aware of how much I have learned because of what responding to the pandemic made me pay attention to. I am a bit overwhelmed by how much I have learned that I feel like I should have known before. I have learned about the nuances of aspects of instruction that I thought I knew but it turns out they are a lot more complicated than I realized. More about that in upcoming Coaching Letters—this one is about a trap that, it turns out, I had fallen into by not paying close enough attention to what really matters in instructional practices that I thought I understood.
I was in California last August for my nephew’s wedding, which was lovely and fun and we had a great time. I did, however, underestimate how quickly I would finish the books I brought with me (which, just so you know, included a Tom Clancy novel), which meant that I was staring at the deeply alarming prospect of not having a book to read on the long trip home. Luckily, they have bookstores in California, and I bought, with no prior knowledge about it, a book called The Book of Why. This turned out to be a goldmine, with lots of useful ideas, but the one that stood out to me the most was the idea of conditioning on the wrong variable.
So, let me explain what this is with a story that turns out to be apocryphal (and humans being the wonderful curious beings that they are, here is a blog post that explores the origin of this urban legend). The story has it that researchers were training a neural network to distinguish between American and Soviet tanks—big armored vehicles with caterpillar tracks and cannon. The way they do this is to input a variety of pictures of tanks, labeling them as being American or Soviet, and the computer makes inferences from this training set as to how to tell the difference between the two. How it does that is a mystery to me. Then the computer is given a fresh set of pictures that it hasn’t seen before, and asked to specify whether the pictures show American or Soviet tanks. And, the story goes, the computer failed at this task. And the researchers, after a suitable period of gnashing of teeth and pulling of hair, realized that all the pictures of the American tanks in the training set were taken on a sunny day, and all the Soviet pictures were of overcast skies. So the computer had inferred that the weather was the determining factor in deciding whether a tank was American or Soviet. But that, of course, is not correct. The computer had conditioned on the wrong variable.
Once you get your head around this idea, you start noticing it all the time. Here are some examples.
Albert Brooks in the Atlantic writes a regular column about happiness called How to Build a Life. He often makes the point that the things that people think are going to make them happier do not have the intended effect. More money is one of them—there may be a temporary bump in your mood if you get a pay raise or win the lottery, but sooner or later you revert back to your previous level of happiness. Here’s my favorite of his essays, explaining why happiness is not about money or stuff. Chasing money in service of well-being is, therefore, conditioning on the wrong variable.
The halo effect is a cognitive bias that causes people to make judgments based on a single trait of a person or thing. It can lead to inaccurate judgments, and can be seen in many areas of life, including education, the workplace, and social interactions. But we do it all the time. We may think that someone who went to an elite university is more knowledgeable than they really are. We may suppose that a billionaire who made their fortune in business has expertise in making government more efficient, when the mission of government is fundamentally different from business. We may assume that if Apple releases a new product it will be high-quality simply because of the company’s reputation, even before using it. These are all examples of conditioning on the wrong variable.
And then of course, there is the flip of the halo effect, when we assume that some characteristics are indicative of unrelated undesirable characteristics, such as assuming that blondes are ditzy, or students of color come from impoverished backgrounds, or girls are not as good at math as boys, and so on. These are biases, or prejudices. And they are definitely conditioning on the wrong variable .
Advertisements for cars are often about features that have nothing to do with what really matters in a car—safety, reliability, efficiency. What the advertisers are asking you to pay attention to is how the car is going to make you feel, or the social status that it will bring, or how the car looks. They are, in other words, trying to get you to pay attention to the wrong variables.
We make the mistake of conditioning on the wrong variable all the time in education. Here are some examples.
When we ask workshop participants to talk about effective feedback, their initial ideas frequently have to do with features such as timely, specific, actionable and personal. But what really matters with feedback—in other words, the right variable—is whether the feedback is actually useful to the learner (whether that learner is a student or an adult). This is not to say that timely and specific are not important, but they are not the critical variables. Paying attention to timeliness over utility is conditioning on the wrong variable.
We still see teacher evaluation rubrics and walkthrough checklists that include having a learning objective written on the board. There are a few parts to this. First, having a learning objective is absolutely essential for the teacher, but does not improve student learning in the way that many educators assume that it does—it is associated with self-regulation, not achievement. Second, whether a learning objective is posted is incidental to whether the learning objective, or the related task, is actually any good. And third, what really matters for students is the thinking that they are being asked to do, not whether they know the learning objective. Indeed, the learning objective may not even mean anything to them until they have actually learned the material in the lesson. Paying attention to whether a learning objective is posted is conditioning on the wrong variable.
We think choice matters, for both adults and students. It doesn’t. (I’ve started writing about choice and autonomy, and you are welcome to look at what I’ve written so far, but then you have to give me feedback. It will likely be a future Coaching Letter. And thanks to those who have given me feedback so far—I have not had a chance to incorporate your comments, but I will.) But having choice is a pervasive idea. For example, in Student Centered Coaching, the authors make the very bold statement “It’s never a good idea to set goals for teachers…” That is based on all sorts of assumptions about the psychology of motivation that are not as cut-and-dried as the authors think, and ignores the organizational prerogative for implementing programs and practices. What matters is that people have clarity about what they’re aiming for, they believe they can attain it, they think the target will stay still long enough, and that there are not costs for giving up what they’re currently doing. Teachers/adults need choice is a myth, and is conditioning on the wrong variable.
My colleague Tom also writes a Substack, focused on research on teaching and learning, especially but not exclusively in math. If you have not already signed up for it, you should do that here. So far, his topics include inquiry v. direct instruction, grouping v. individual work, and cognitive load. His most recent post is about the number of students who are now working at vertical non-permanent surfaces (VNPSs, à la Building Thinking Classrooms), and how that does not mean that they are doing more thinking. Assuming that a thinking classroom is indicated by kids working at whiteboards is conditioning on the wrong variable. There’s another great post about that: It’s Not About the Boards.
Finally, some quick announcements. Last call to sign up for the Next Level Instructional Leadership workshop next month in western Massachusetts. We ran this twice already, in Connecticut and Rhode Island, and it’s been fabulous, even if I do say so myself. Please come if you can! Also, a few seats left at the In-Depth on May 15 at Mercy, focused on High School. And if you haven’t already seen it, our Instructional Leadership Collective is featured in the latest issue of PEL’s newsletter. Please check it out! I love this group of central office instructional leaders and look forward to the meetings enormously. If you know someone who might be interested (teams are best!) please send them my way.
Thank you again for reading The Coaching Letter. Please let me know if there is anything I can do for you. Best, Isobel