Nick Kristof is not Smarter than an 8th Grader

About a week ago, Nick Kristof published this op-ed in the New York Times. Entitled Are You Smarter than an 8th Grader, the piece discusses American kids’ underperformance in math compared with students from other countries, as measured by standardized test results. Kristof goes over several questions from the 2011 TIMSS (Trends in International Mathematics and Science Study) test administered to 8th graders, and highlights how American students did worse than students from Iran, Indonesia, Ghana, Palestine, Turkey, and Armenia, as well as traditional high performers like Singapore. “We all know Johnny can’t read,” says Kristof, in that finger-wagging way perfected by the current cohort of New York Times op-ed columnists; “it appears that Johnny is even worse at counting.”

The trouble with this narrative is that it’s utterly, demonstrably false.

My friend Jordan Ellenberg pointed me to this blog post, which highlights the problem. In spite of Kristof’s alarmism, it turns out that American eighth graders actually did quite well on the 2011 TIMSS. You can see the complete results here. Out of 42 countries tested, the US placed 9th. If you look at the scores by country, you’ll see a large gap between the top 5 (Korea, Singapore, Taiwan, Hong Kong, and Japan) and everyone else. After that gap comes Russia, in 6th place, then another gap, then a group of 9 closely bunched countries: Israel, Finland, the US, England, Hungary, Australia, Slovenia, Lithuania, and Italy. Those made up, more or less, the top third of all the countries that took the test. Our performance isn’t mind-blowing, but it’s not terrible either. So what the hell is Kristof talking about?

You’ll find the answer here, in a list of 88 publicly released questions from the test (not all questions were published, but this appears to be a representative sample). For each question, a performance breakdown by country is given. When I went through the questions, I found that the US placed in the top third (top 14 out of 42 countries) on 45 of them, the middle third on 39, and the bottom third on 4. This seems typical of the kind of variance usually seen on standardized tests. US kids did particularly well on statistics, data interpretation, and estimation, which have all gotten more emphasis in the math curriculum lately. For example, 80% of US eighth graders answered this question correctly:

Which of these is the best estimate of (7.21 × 3.86) / 10.09?

(A) (7 × 3) / 10   (B) (7 × 4) / 10   (C) (7 × 3) / 11   (D) (7 × 4) / 11

More American kids knew that the correct answer was (B) than Russians, Finns, Japanese, English, or Israelis. Nice job, kids! And let’s give your teachers some credit too!

But Kristof isn’t willing to do either. He has a narrative of American underperformance in mind, and if the overall test results don’t fit his story, he’ll just go and find some results that do! Thus, the examples in his column. Kristof literally went and picked the two questions out of 88 on which the US did the worst, and highlighted those in the column. (He gives a third example too, a question in which the US was in the middle of the pack, but the pack did poorly, so the US’s absolute score looks bad.) And, presto! — instead of a story about kids learning stuff and doing decently on a test, we have yet another hysterical screed about Americans “struggling to compete with citizens of other countries.”

Kristof gives no suggestions for what we can actually do better, by the way. But he does offer this helpful advice:

Numeracy isn’t a sign of geekiness, but a basic requirement for intelligent discussions of public policy. Without it, politicians routinely get away with using statistics, as Mark Twain supposedly observed, the way a drunk uses a lamppost: for support rather than illumination.

So do op-ed columnists, apparently.

Advertisements

7 thoughts on “Nick Kristof is not Smarter than an 8th Grader

  1. Pingback: Another Kristof Krissis – TIMSS results used and abused | mindstorm tools

  2. A “national columnist” being sloppy with numbers is also an abuse of his presumed position as a media figure and an “authority” (irony intended). When this happens — and it happens too often — I can’t help wondering how much difference there really is between the NYT and Fox News

    Like

  3. Tara Hollingsworth

    The difference is that Fox News TRIES to be fair and balanced. The NYT has completely surrendered to the Dark Side.

    Like

  4. Kristof was writing mostly about American adults and the ETS report (http://www.ets.org/s/research/30079/overview.html). This report clearly demonstrates the grim status of the literacy of our millennium generation. Figure 1 of this report shows that among the 23 participating countries, American adults rank 17th in literacy and 21st in numeracy. And among the 20 participating countries, they ranked 18th in problem solving in technology-rich environments (PS-TRE).

    8th-grade results are not as important as 12th-grade’s. TIMSS 2015 will be an important barometer to watch. We can predict what will happen based on the 12th graders’ report card: 74% of students are below the proficient level! (http://nces.ed.gov/nationsreportcard/subject/publications/main2013/pdf/2014087.pdf)

    Cheering the small positive and ignoring the big negative is not going to help our educational crisis. See this Washington University’s research on “carrot and stick.” (http://www.eurekalert.org/pub_releases/2015-05/wuso-cos050615.php)

    Like

    • When Kristof published his column, he tweeted (https://twitter.com/NickKristof/status/592314195628003328) this: “My column notes that US kids lag the world in basic math.” The column is called “Are You Smarter than an 8th Grader?” I’m sorry, but the focus is kids, not adults.

      As you say, the ETS report is about the PIAAC survey of ADULTS’ math skills. This brings in a lot more factors than just school quality, and it’s very misleading to use one as a proxy for the other. It feels like Kristof talks about PIAAC so he can say the US came in last. But even that last place standing is misleading, because the population surveyed is different: advanced countries only (22 of them) in the case of PIAAC, a much broader range (42 countries) in the case of TIMSS (the 8th grade test). Kristof doesn’t say it, but the column is constructed to imply that American school kids did worse than kids from Iran and Ghana on a standardized math test of 8th graders, and that’s not true in any reasonable sense.

      Like

  5. Pingback: Raj Chetty Discovers that Poverty Matters After All - Living in Dialogue

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s