Education's World Cup? Not exactly ... We'd do well to take our slice of the Pisa with a large pinch of salt

News | Published in TES Newspaper on 10 December, 2010 | By: Peter Wilby

• Comment:

• Last Updated:

  16 December, 2010

• Section:

  News

Education now has its own World Cup, albeit at three-yearly intervals, rather than the four years favoured by most major sports. The results of the latest contest held in 2009 - the Programme for International Student Assessment (Pisa) - have just been announced to the usual political and media excitement. And it is not only in Britain that results are awaited and analysed with anxious attention. Across the developed world, the Pisa league tables influence public attitudes to schools and government policies on curriculum, assessment and governance.

Finland, which came top in the first three contests (2000, 2003 and 2006), became, to its astonishment, a place of educational pilgrimage. Education ministers from across the world demanded to know its secrets. It has become to education what Australia is to cricket, Brazil to football and Kenya to long-distance running. At the other end of the scale, Germany was so devastated by its 21st position in 2000 - below such neighbours and rivals as France, UK, Austria, the Czech Republic and Italy - that, after a prolonged bout of soul-searching, it introduced national tests of learning outcomes in core subjects. It may argue that its stronger showing in this year's tests is a result of the changes.

But there's a big difference between Finland and Germany. The first is a small country on the fringes of Europe, with few social and ethnic divisions. Germany, on the other hand, is a large country in the centre of Europe, with a big Turkish minority. Might this have something to do with their respective scores? Immigrants don't always depress results but, where they come from poor countries and/or remote rural regions and suffer substantial prejudice, segregation and discrimination, they tend to do so. When results were re-analysed to look at native-born children only, Germany's position improved sharply.

There are other things about Finland. The Finnish language contains no irregular words and each letter makes a single, logical sound. So, it is argued, Finns learn to read more easily than, say, Britons or Germans. Another suggestion is that the harsh climate and long periods of foreign rule (by Sweden and Russia) have created a stoic character among its people - which, presumably, makes Finnish children more inclined to sit quietly in class and buckle down to homework.

That's the trouble with international comparisons. We all know the arguments about league tables for English schools. What allowances should be made for home background? Should children with special needs be included? How do we account for absentees? How can we stop teaching to the test or outright cheating? These questions are also debated endlessly - sometimes in forthright language - over the Pisa tests which are run by the Organisation for Economic Co-operation and Development (OECD), a sort of mini-UN for the developed world. Other issues enter the equation. Depending on assessment systems in their own countries, some pupils may be more practised than others in written test techniques. Korea, which came top of the OECD countries this year, has a regimented education system and its pupils train hard for the tests. Some researchers claim to have found national differences in the seriousness with which children approach tests. There are problems, too, about the samples used. Austria, for example, was alarmed to discover its league table position had slumped in 2003 compared with 2000. Only then did it find that the 2000 sample underweighted vocational schools and the decline was therefore illusory. Similarly, the UK's excellent results in 2000 turned out to be partly attributable to an over-representation of independent schools.

The truth is that Pisa cannot offer a guide to the relative success of different school systems. It tests 15-year-olds on the "cumulative impact of learning ... experiences both in school and at home". The tests are not about knowledge and understanding of school subjects as conventionally defined; they are about what Pisa rather confusingly calls "literacy" - in maths and science, as well as in the native language - meaning the ability to use these subjects in everyday life. The results do not correlate well with those from another testing regime, the Trends in International Mathematics and Science Study (Timms), which does gear its tests to subject knowledge. Hungary comes near the top in Timms, but well down the Pisa league; for New Zealand, it's the other way round.

Pisa, you may say, measures what we want from schools - not academic theory, useful only in the classroom, but practical skills that children carry into adulthood. Unfortunately, it is impossible to tell whether the latter were learned inside or outside school. According to Alan Smithers, professor of education at Buckingham University, pupils who haven't studied science often do better on the Pisa science tests than pupils who have. As a world cup tournament for education, Pisa makes about as much sense as a football world cup that awards points for fencing, archery and tiddlywinks. Smithers and other researchers point out that Pisa is merely a "snapshot" study; it offers no longitudinal data, following children's progress through school, that would allow a value-added measure, similar to that used in the English league tables.

None of these imperfections will deter politicians and journalists from continuing to draw conclusions from the tests, usually in line with their preconceptions and ideological prejudices. In the neoliberal era, governments can no longer intervene directly to create economic growth. Their job is to tax and regulate as little as possible. Their responsibility, as they see it, is to oversee education systems that provide the population with the skills to secure "competitive advantage". But if the Pisa results are supposed to be a measure of success in that aim, they fail miserably. The world's richest country, the US, does badly in all international educational comparisons, while the top two in GDP per capita - Norway and Luxembourg - struggle in the lower echelons of the Pisa league.

Peter Wilby is a former editor of the New Statesman.