Skip to main content
article icon

Pisa rankings are 'utterly wrong'

news | Published in TES magazine on 19 July, 2013 | By: William Stewart

Experts say global league tables beloved by politicians are ‘useless’

The world’s most influential education league tables are “useless”, produce “meaningless” rankings and are compiled using techniques that are “utterly wrong”, according to damning new academic criticism.

Politicians around the globe are increasingly using Pisa (Programme for International Student Assessment) results to formulate and justify their school reforms, often focusing on the headline country rankings. But researchers are now raising serious concerns about their reliability.

In response, the Organisation for Economic Cooperation and Development (OECD), which runs Pisa, has admitted that “large variation in single (country) ranking positions is likely”. But the academics that TES has spoken to argue that there are even deeper problems that the body is not acknowledging.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, said that an inappropriate model is used to calculate the Pisa rankings every three years. In a paper published this summer, he challenges their reliability and shows how they fluctuate significantly according to which test questions are used. He reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Dr Hugh Morrison, from Queens University Belfast in Northern Ireland, goes further, saying that the model Pisa uses to calculate the rankings is, on its own terms, “utterly wrong” because it contains a “profound” conceptual error. For this reason, the mathematician claims, “Pisa will never work”.

The academics’ papers have serious implications for politicians, including England’s education secretary Michael Gove, who justified his sweeping reforms by stating that England “plummeted” down the Pisa rankings between 2000 and 2009.

The questions used for Pisa vary between countries and between students participating in the same assessment. In Pisa 2006, for example, half the students were not asked any reading questions but were allocated “plausible” reading scores to help calculate their countries’ rankings.

To work out these “plausible” values, Pisa uses the Rasch model, a statistical way of “scaling” up the results it does have. But Professor Kreiner says this model can only work if the questions that Pisa uses are of the same level of difficulty for each of the participating countries. He believes his research proves that this is not the case, and therefore the comparisons that Pisa makes between countries are “useless”.

When the academic first raised the issue in 2011, the OECD countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point. But Professor Kreiner’s new paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis.

He is sceptical about the whole concept of Pisa. “It is meaningless to try to compare reading in Chinese with reading in Danish,” he said.

Dr Morrison said that the Rasch model made the “impossible” claim of being able to measure ability independently of the questions that students answer. “I am certain this (problem) cannot be answered,” he told TES.

The OECD’s response did not address Dr Morrison’s claims, but largely restated its arguments against Professor Kreiner’s original 2011 paper on Pisa.

“Large variation in single ranking positions is likely, particularly among the group of countries that are clustered in the middle of the distribution, as the scores are similar,” the organisation said. It claimed that country rankings take account of “the uncertainty that results from sample data” and are “therefore shown in the form of ranges rather than single ranking positions”.

These ranges are given in Pisa results but not in the main tables. And although separate rankings are produced for England, no ranges are published for the country.

The OECD said Pisa questions were “tested to ensure they have the same relative difficulty across countries”, but it has admitted to TES that some variation remains after this testing. On the suitability of Rasch, it said “no model exactly fits the data”.

“Examination of alternative models has shown that the outcomes are identical,” it added. “However, Pisa will always seek to improve and learn from the latest research methods.”

See next week’s TES for a full report on the Pisa concerns.

 

Photo credit: Getty

Original headline: Pisa rankings are ‘utterly wrong’, academic claims


Subscribe to the magazine

3 average rating

Comment (5)

  • The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. "Sir, why did they ask about dinosaurs and geology when we don't do it?"

    Unsuitable or offensive? Report this comment

    Rating: 5 out of 5 stars
    8:49
    19 July, 2013

    cochrane1964

  • The most telling line of this myth-shattering story is William Stewart's reporting that the usually loquacious and acerbic PISA have no reply to the charge of using a model containing a conceptual mathematical error. The OECD's response did not address Dr Morrison’s claims.Also surprising is the absence of an attack paper from Pisa's go to academic Professor Raymond Adams in the manner in which he criticised S J Prais and Svend Kreiner for having the temerity to raise concernson PISA.
    How out of character for Andreas Schleicher and his team. I trust he now making plans to return the monies collected from the 67 participating member states.

    Unsuitable or offensive? Report this comment

    10:42
    19 July, 2013

    Stevemayman

  • The report’s conclusion – that we should abandon the attempt to compare our performance against the rest of the world – is silly and self-defeating. When the Pacific Rim abandons economic competition with the west, one might consider it, but until then we must look for all the clues we can find to raise our game.

    No test is watertight or perfect, and in any case, the rank order is the least interesting aspect of PISA. For useful information, go to the attitudinal surveys and the profile of performance that show where we lead the world and where we trail it. There are also clues there about how we need to change to improve our standing.

    It’s no bad thing to challenge the way PISA results are calculated but I can think of easier ways to influence Andreas Schleicher’s methodology than strirring up a press frenzy that draws attention away from the findings.

    Unsuitable or offensive? Report this comment

    13:32
    19 July, 2013

    SueHackman

  • What Pisa studies also do not take account of is the fact that the demographic spectrum of schools and classes differ widely across countries and regions. Some years ago I was conducting a research study which included testing 7 year olds for their general reasoning abilities. When carefully explaining how to complete a multiple choice reasoning test produced by NFER to 32 7 year olds in a London primary school, after 10 minutes, the teacher whispered "don't forget, 80% of them don't speak any English!" This meant that not only would they struggle with their English lessons and assessments but that they struggled with learning science and maths because the booklets were in English. When the media compare the Pisa results of the UK with Japan or Finland no consideration is given to the fact that in Japan and Finland 99% of the pupils completing the tests are native language speakers, whereas it can be less than 50% in many UK urban schools. No comparison.

    Unsuitable or offensive? Report this comment

    10:59
    22 July, 2013

    MargaretJCox

  • If Andreas Schleicher was an academic he would have been on the VC's carpet to explain this: http://www.scienceguide.nl/201308/schleicher%E2%80%99s-quantitative-counterattack.aspx
    The TES management don't seem to care about copyright in the same way Schleicher doesn't care that OECD Pisa's rankings model is "utterly useless".
    There isn't one shred of quantative counterattack in this rehash of the Aug 2nd article.

    Unsuitable or offensive? Report this comment

    18:36
    14 September, 2013

    Stevemayman

Add your comment

Subscribe to the magazine

Related articles

More Articles

Join TES for free now

Join TES for free now

Four great reasons to join today...

1. Be part of the largest network of teachers in the world – over 2m members
2. Download over 600,000 free teaching resources
3. Get a personalized email of the most relevant resources for you delivered to your inbox.
4. Find out first about the latest jobs in education

Images