Skip to main content
article icon

Gove accused of building on shaky Pisa foundations

news | Published in TES magazine on 2 November, 2012 | By: William Stewart

Doubt is cast on international data he cited to drive reforms

Michael Gove has made England's apparent plunge down the international education league tables the central justification for his sweeping school reforms. But now the government's own statistics watchdog has called into question the figures at the heart of the education secretary's argument.

The UK Statistics Authority has censured the Department for Education and Sir Michael Wilshaw - appointed by Mr Gove as Ofsted chief inspector - for using uncertain, weak and "problematic" statistics to claim that England's schools have tumbled down the global rankings.

In a television interview in September, Sir Michael asserted that "our standards are falling in relation to other countries in the rest of the world", citing a significant drop in England's Programme for International Student Assessment (Pisa) ranking between 2000 and 2009.

The DfE used the same figures in a 2010 press release, claiming: "England has continued to fall in the Pisa rankings, meaning that in just nine years we have dropped from 7th to 25th in reading, 8th to 27th in mathematics and 4th to 16th in science."

Now Andrew Dilnot, UK Statistics Authority chair, has written an open letter criticising Sir Michael and the department, noting that the Organisation for Economic Co-operation and Development (OECD), which compiles Pisa, no longer views the UK results from 2000 as statistically valid because not enough schools took part.

Mr Dilnot also cites a review by the University of London's Institute of Education, which highlighted the statistical problems and the contradiction of Pisa findings by the rival Trends in International Mathematics and Science Study.

The institute's report concluded that "it may be difficult to treat an apparent decline in secondary school pupils' performance as 'a statistically robust result'", according to Mr Dilnot.

"Given these various published caveats and advice, my own view is that those commenting on data in this area should take particular care to avoid making comparisons that could be interpreted as statistically problematic, or otherwise might be misunderstood by the audience," he writes.

His verdict will be seen by critics as a blow to Mr Gove's claim that England has "plummeted in the world rankings" given that the education secretary has been so unequivocal about the figures, arguing that "these are facts from which we cannot hide".

Andreas Schleicher, the OECD official who runs Pisa, told TES that the data used by Mr Gove was "a little bit dodgy". But he added: "The more important lesson for me is that the UK has not improved in a way that we have seen other systems improving."

Mr Dilnot's letter is addressed to Labour MP David Miliband - schools minister between 2002 and 2004 - in response to a complaint by the MP about Sir Michael's use of the Pisa figures.

The statistics watchdog chair also goes on to criticise the DfE press release, writing: "Headline results for England from the Pisa study, alongside relative international rankings, were not accompanied by detailed advice or caveats to help the reader in making comparisons over time, nor were the statistical implications of an increase in the number of reporting countries in later Pisa studies noted.

"The weaknesses relating to the response-rate standard in earlier studies should not be ignored."

Mr Dilnot's letter adds: "These uncertainties and weaknesses are not just a technical footnote; they are themselves an important part of the evidence, and affect interpretation and meaning.

"League tables and the presentation of international rankings can be statistically problematic, and require clear and careful commentary alongside them. I will find a suitable opportunity to explore these matters further with the department."

Labour shadow education secretary Stephen Twigg said: "Michael Gove's department needs to be very careful in its use of official statistics, especially at a time when he has undermined the teaching profession. Running down our teachers and schools is not the best way to raise standards."

An Ofsted spokeswoman said: "The interpretation of international comparisons is complex, but the overall message from Pisa is clear. England's mean scores fell, particularly between 2000 and 2006, while other countries in the survey improved. There are also concerns in light of the latest 2009 results, where many other countries are noticeably outperforming England."

The DfE had not provided a comment at the time of going to press.


Subscribe to the magazine

as yet unrated

Comment (4)

  • At last. It's been nearly two years since FullFact highlighted the warning by OECD that the 2000 PISA scores for the UK were unreliable. FullFact's campaign about the misuse of this data has at last been recognised. David Cameron, Michael Gove, Nick Gibb, Lord Hill, all the Tory MPs who quoted the figures so confidently, Sir Michael Wilshaw, most of the media (including Toby Young) should now apologise for misleading the public.

    I'm not holding my breath. When an Ofsted spokeswoman can still cite England's score from 2000 in a response to a question about the unreliability of the 2000 figures, it makes me wonder whether anyone in Government and its tame quangos have basic comprehension skills. Or perhaps they think the electorate is stupid and can be easily fooled - just quote a few international statistics at the plebs and they will believe anything.

    This is my message to them in words of one syllable: The 2000 PISA scores for the UK are flawed - don't use them.

    http://fullfact.org/factchecks/school_standards_oecd_pisa_data_media_conservatives_education-2423

    Unsuitable or offensive? Report this comment

    13:09
    2 November, 2012

    Retired123

  • I guess the SPADs who came up with the ruse are now getting it in the neck - interesting to see what happens next ...

    Unsuitable or offensive? Report this comment

    14:10
    2 November, 2012

    eyebeams

  • For a summary of problems with PISA and links to more detailed academic criticisms see Stephen Heppell's page http://www.heppell.net/pisa/

    Gove use of unexamined PISA headline stats seems very much less than 'rigorous' (to use one of his own highly favoured terms).

    Unsuitable or offensive? Report this comment

    16:03
    7 November, 2012

    Alex Jones

  • So we have lies, damed lies, statistics and then Gove's use of statistics. Sadly this is not new news and I'm sure nothing will happen from it - is it time for another enquiry? Of course not.

    Unsuitable or offensive? Report this comment

    10:14
    11 November, 2012

    paulhopkins

Add your comment

Subscribe to the magazine

Related articles

More Articles

Join TES for free now

Join TES for free now

Four great reasons to join today...

1. Be part of the largest network of teachers in the world – over 2m members
2. Download over 600,000 free teaching resources
3. Get a personalized email of the most relevant resources for you delivered to your inbox.
4. Find out first about the latest jobs in education