Lies, damn lies, and English education statistics.
This article was first published on the Academia Libera Mentis Substack.
On September 8th, the PISA results came out, detailing how almost a million 15-year olds across the world performed in standard tests for reading, mathematics, and science. The PISA results have become more and more important since they were set up in 2000 to track the progress of education in rich countries – the countries of the Organisation for Economics Cooperation and Development (OECD).
One overall picture of education over time is in the graph below.
This graph shows the scores on reading, which is perhaps the easiest metric to compare across countries, over time for the ‘10th percentile’. The 10th percentile is the student who is 10th from the bottom – or 90th from the top, so a relatively low performer. That student represents the bottom of the schooling system. For the OECD as a whole, you can see that performance has dropped markedly since 2015, with a quite dramatic reduction in 2022 and 2025. We will talk in later episodes about reasons for that, but here we just note the strong decline in all countries.
Well, all countries? Not quite…
At ALM we are critical of data. We always worry about the people gathering the data: what incentives do they have, how do they interpret the world, what do they actually do?
A good rule of thumb to discover data manipulation is to look at something that seems too good to be true and delve into that case. For the 2025 PISA results, the ‘too good to be true’ prize must go to England. The 10th percentile in England has performed a lot better in 2025 than in 2022. Look at how well the students in England did relative to those on the same island and roughly same system, like Wales:
Note here that we again have England ‘doing well’, this time on average rather than just the 10th percentile. And note how Wales, on the same island, mainly under the same government and education system, kept dropping after 2022, along with the OECD average. England is the outlier to a quite spectacular degree. Fishy, no?
England is now even in the top 10 of the OECD rankings, an unheard of ‘achievement’. So of course you have the current government proudly proclaiming this success is due to their recent policies, just as the previous government is claiming it is due to earlier policies finally paying off. Such is inevitable in politics.
The truly dispassionate notes that in the same period 2022-2025 there have been a lot of highly negative noises about education in England. A supposed flood of poorly educated young migrants (which you see in the England PISA 2025 results where the first generation migrants are doing worse than others). A supposed feminisation of teachers, and an invasion of both woke ideology and boy-bashing rules designed to eliminate the last vestiges of masculinity in the classroom. You thus see increased rates of medicine use, special needs students, transgenders, obesity, and English as second language. Are we really to believe that despite these developments English students are doing relatively much better than before? Nah, if results are so counterintuitive, one should first suspect data manipulation: the onus of proof shifts to those making the extraordinary claims.
Before I tell you of what I found and deduced, note that you can test your level of statistical literacy here with a real data puzzle that is only two days old. Would you be able to spot data manipulation surrounding England 2025 PISA results yourself using any available sources? If you can’t, you should realise the value of an ALM education. No kidding: being able to spot whether you are being told the broad truth or an unwarranted assertion when it comes to how education in a country is going is a near perfect test of whether you are truly capable of spotting misinformation yourself or whether you still need continuous help and thus you are not actually capable of learning from data at all. So, give it a go and be honest to yourself: are you able to see through the data manipulation in this case or not?
Let’s then talk about the steps undertaken to see what was going on. How would anyone know whether the schools and education authorities in England have manipulated the data? Of course, a first instinct many have is to first ask AI, which I did. Google AI duly reported that nothing is amiss and England has achieved a wonderful success. That is what AI (roughly) said within two hours of releasing the PISA reports no less, which suggests a wonderful marketing effort behind the scenes to pre-empt criticisms. Other AI was not much different.
But we don’t trust AI at ALM with anything politically or commercially sensitive, so I dug deeper. In order to minimise the effort I had to make to get onto the right trail, I asked AI who had been critical of the previous PISA results in England in 2022. I got a name: Professor John Jerrim. Ok, then I looked up what he said just after the release of the 2025 results (I put his name into X). He tweeted the Table below, which is a table appearing on page 184 of the England-specific report, meaning you will not find it in the 377 page main report of the OECD, but you really have to go the government webpages of the UK and then fork out the England report on its website. I could have found that out myself, but Professor John Jerrim provided me a short-cut to where the skeletons were hidden. Thanks Professor Jerrim!
I started hunting and reading data. The key data in the report that John noted are in the table below, which gives two main indications that, indeed, the England PISA results are completely unreliable.
For one, look at the last row of this table, saying that 1.4% of the participating students had an ‘Education, Health, and Care plan’ (EHC), versus 7.1% of the non-participating students. Below the table it is clear that there were 5,209 participating students, 1,651 non-participating ones, and 410 students who could not be matched. Ok, what do these two bits of information tell us? Well, EHC students will be the worst performing students: those are the ones who can’t sit still, have severe learning difficulties and have major health problems. They will normally score very low. In the group of 1,651 students who did not do the final PISA tests, 7.1% was in this low-performing group, five times more than the 1.4%: the final group doing the PISA tests has far fewer of the low performers than the group that for ‘some reason’ did not do the final test. Convenient, no?
What about those 410 students for which ‘NPD matches’ were not possible, who will those be? Well, matches are needed between student data and national data, which was not possible for 410 students. The kind of student that cannot be matched will include a lot of recent migrants, students who switched schools a lot, or who changed names recently. All indicators of low performing students. Convenient that 410 of those were not in the final data, no? And note that in most other OECD countries, there is a national population register that will make matching much easier than in the UK, meaning that other countries will have a lot less bias. Again, convenient.
Of course, one then does a background check. Does the 1.4% line up with national data? No. It turns out that in 2025 5.3% of UK school children have EHC plans, hence more than three times more than in this PISA data. Something similar holds for the broader category of children who have difficulties: those with SEN support, where SEN means ‘Special Education Needs’. The country has 19.5% of those in schools, whilst the final sample in the PISA only has 10.7% (and the non-participating sample 25%). So the English schools in the PISA see a very heavy selection against poorly performing kids. Basically most of the bad performers were not doing the tests, which is exactly what you’d expect from our two graphs above: the bottom 10% of the English students have done particularly well in 2025, not just relative to other countries, but also relative to the average in England (which did not improve!). Perchance the teachers told the worst performers it was time to be sick on the days of the test? It is an old trick, but it works.
A final preliminary test then: how does the bottom in England match up against other countries? It turns out that the bottom 25% in England did as well as the middle persons in Wales. And in Wales, if I look for selectivity of a poorly performing group, I find on page 176 of their report that the proportion of particularly needy students (called ALN in Wales) is 8.6% for the final group of participating schools, compared to about 10% for Wales as a whole, which is a much smaller difference than 1.4% versus 5.3%. Hence, either Wales is full of particularly poor performing students, or the English data is particularly massaged at the bottom. No prizes for what seems most likely!
The scent is picked up!
With the above in mind, I started engaging a bit online, quickly finding myself in a battle with the defenders of the English school system – see as with Stuart Lock, the ‘chief executive’ (not a teacher!) of ‘Advantage Schools’, eg.:
These defenders of the PISA England 2025 results apparently did not want to see anything wrong. Yet, I was also quickly followed by other groups within the system that had other interests, including the Welsh Treasury! When such things happen, you know you are onto something. So I decided to look deeper into the issues.
To start with, I dug a bit more into that England-specific report, finding a few very fishy things indeed.
For one, I asked myself who actually put the data together. In several countries it is a national statistical agency or something close to that. Those are usually the best ones because they have a lot of reputation at stake across a huge number of statistics, and of course they have a lot of professional statisticians in their employ.
In the UK, although the final reports comes from the Department of Education, the data was gathered by Pearson, a private company. Oh oh. Worse, Pearson is a private company that sells a lot to education authorities in England (about 90% of the UK population is in England, so the English are the client to please, not the 10% in Wales and Scotland). That is a major red flag: the people gathering the data cannot reasonably be considered an uninterested party. The dangers of having a private company gather sensitive data for a government department and whole system it does other work for is of course widely known in the literature on corruption (on which I have written two books): a lot of dirty work is by companies that do deals with governments. Deniability and defamation laws make it an attractive avenue. Of course I wouldn’t dare to accuse Pearson of anything, just noting possible motives known from the literature.
Secondly, I crawled a bit more into the data the report provides at the back about sample selection of schools and pupils.
I noted that 24.1% of the participating schools had a missing ‘attainment band’ (Table A5), whereas in principle every participating school should have one. This is a huge degree of missing data, always a red flag if one is looking for data manipulation because the person setting up the data can play with that missing category, such as by selecting top schools and pretending their attainment band is unknown (which you can always justify by saying it is not known for some particular year). Also possible is to rummage deliberately in the set of schools with missing attainment bands and select the ones you like. The 24.1% thus suggests an enormous amount of discretion on the part of the commercial company doing the data gathering. I note that in Wales only 9% of the PISA schools had a missing attainment band (the unnumbered table on page 173).
Then I noted something similar with the students. It is the case that the participating schools in the final sample have better students than those who dropped out of the sample (Table A.4), that these schools did better than the average in England as a whole, and that the better schools had more pupils doing the actual PISA tests than the less good schools. How can you see this? Well, you look for the national average from the government itself and compare it with what is in the PISA 2025 report. In the latest reported data from GOV.UK (for 2022/2023), 45.4% of pupils achieved a grade 5 or higher in both English and Maths; versus 47.1% with grades at least 5 in the weighted final PISA sample. It is not a huge difference, but a significant shift somewhat on top of the EHC selectivity.
Let us then do a bit of ‘magical’ rule-of-thumb statistics to see how big the effect of selectivity might have been. If I interpret the 1.4% versus 5.3% EHC selectivity as a 3.9% shift upwards in the distribution of students, and add the 47.1% versus 45.4% selectivity as a 1.7% further shift upwards, I get a 5.6% upwards shift in the distribution, basically as if I took out the worst 6 students of every 100. If you then note in the report itself that the range between the 10th percentile and 90th percentile in science is 264 (averaged over the genders), meaning that an 80% shift in the distribution is worth 265, then that 5.6% shift is worth1 18.55 points in those PISA scores. An 18 point shift downwards would put England on the same level as Scotland and in line with the OECD trend.
Are there reasons to suspect that more than 18 points might be involved in the sample selectivity? Yes, indeed. Whilst the above is about selectivity of the bottom students, there is also the question whether the data involved particularly good schools that shifted the top. In this regard, I note that the 90th percentile results in England show a spectacular increase in science test results (making them better even than 2018!), though not an increase in reading and mathematics results for that same 90th percentile. Very fishy. My suspicious mind then wonders how one could smuggle in a few top science schools into the sample. If I take out the additional 10 points specific to the miraculous increase in (science) results between 2022 and 2025, on top of taking out the 18.55 points for sample selectivity at the bottom, then the schools in England suddenly look just as good as …. the schools in Wales. Well, I never! Isn’t that a coincidence?
With these strong statistical indications of data selectivity in hand and some evidence that they indeed could explain the too-good-to-be-true results for England, I start hunting for mechanisms: how does this work at the level of the schools involved? And why am I not seeing the same thing in Wales and Scotland, part of the same island: why would schools in England massage their tests whilst schools in Scotland and Wales do not?
Massaging tests as a bureaucratic culture?
I first off asked myself if there was any evidence that schools in England in the past have ‘massaged’ the national tests. In pursuing this question, I have the advantage of local knowledge: my own daughter went to a school in London and got her GCSE-levels there. She was a star student already and supposed to do her A-levels in April and May 2020. However, COVID-19 happened and the national exams were cancelled. Indeed, school was cancelled almost entirely, meaning months less tuition.
What did the schools then do in order to give their pupils A-level grades? They basically asked the teachers what was fair to give the students. As you can imagine, the teachers were very generous. My daughter got 4 A*s, which is the very highest one can possibly get. She was pleased with the grades, thank you very much, and of course she thinks she deserved them, but she would have liked to show her prowess in actual exams.
My daughter was not the only one to get top grades in 2020. Indeed, there was a veritable explosion of top grades that year, not seen before or since. The basic data is quite stark: before 2020, the percentage of students achieving the highest grades in the A-level tests (an A*) was 9% or lower for a decade. In 2020, this jumped to 14.4%, then 19.1% in 2021, 14.7% in 2022, and then below 10% again when the national exams were once more enforced (rather than teacher assessments). Three whole years of magical and unbelievable ’achievements’, right in the period that schools were closed for many months. Who knew that not educating students would be so good for their grades?
As was widely reported in the British press, and subject to several high-level inquiries, many sceptical voices doubted the legitimacy of the “all time high” exam results in England. Schools across the board had ‘allowed’ their students to get top results in record numbers, to the benefit of their students but to the detriment of exam integrity. Indeed, in desperation, the body responsible for exam integrity tried to use AI to adjust exam results downwards, only to be overruled by the politicians after the predictable uproar by schools and parents about how unfair it was that their suddenly-brilliant students were not recognised for the little geniuses they of course were (see: https://oecd.ai/en/incidents/2020-08-13-c27b). Indeed, the percentage of students obtaining a grade (rather than, for instance, not doing the tests because the teacher told them not to show up), also jumped in those covid years, from 96% to 99% and then back again when the true exams returned. I am tempted to think that schools had a more streightforward way to game the system in those years and so did not need to disadvantage their own poor performing students. It was the great covid A-level swindle, lasting for three years.
Ok, so schools in the UK indeed have shown a willingness to ‘report improbable exam results’ when the opportunity presented itself. Note though that in the COVID-19 case, individual teachers could help individual students, which made sense for teachers. But why would English schools then in particular do this again in 2025 for the PISA and not schools in Wales and Scotland? Whatever the reason, it was not simply due to a general policy of the data company: Pearson also collected the PISA 2025 data in Wales and Scotland.
If ‘data massaging’ is especially normal in England, it would have to be normal for the national tests held in England: those are the tests that matter for students and schools.
Consider then the easiest way for an individual school to ‘massage the results’ in a yearly national test cycle, where the test itself is shared with all other schools and out of the hands of the school (as with the PISA tests): you tell the poor performing students not to show up for the exams, or you find some other way to exclude the students whom you know will do poorly. This is indeed what the national data tells us happened before and after the covid years: about 3% fewer students getting grades (note there is no such effect for the subject ‘Welsh’: both before and after covid almost 100% of students in those classes took the exams).
Crucially, it takes time for a school to learn how to do this. Teachers in the school have to become part of this ‘deception’ because they are the ones who know which students will likely do badly and which, even among the EHC and SEN students, will actually do well. Also, teachers are the ones with the contacts with students and parents, thus able to tell the right students not to show up. In principle, teachers will be reluctant to do this because they are disadvantaging their students by telling them not to show up (those students will thus not have grades and not even have the chance to pass!). Only when teachers think it really is in the interest of the school will they do this, which is the kind of habit that needs time to get established and normal.
So, is there any particular reason why schools in England would have gotten used to gaming the exam system, but not schools in Wales and Scotland?
Yes there is: the league table system in which the results for the exams are revealed for every school. Those league tables matter for parents choosing schools, and thus they matter for the school as a whole. Indeed, data massaging is so much part of the landscape in England, that schools are openly and officially warned not to do it (see this BBC report). Worse, there is an academic literature trying to cope with the bad data arising from this league-table driven system.
The crux is that Wales and Scotland opted out of league tables just after the year 2000, whilst schools in England have been doing them ever since 1993. That does explains why students doing Welsh were always doing the exams, both before and after covid: schools in Wales had no pressing reason to game the tests and disadvantage their own students.
We thus have found a smoking gun (various forms of sample selectivity), motive (commercial incentives, schools wanting to look good), and opportunity (schools can game the system), but only for England, not Wales and Scotland. Of course it is not conclusive proof of anything deliberately orchestrated, but one would expect a guilty verdict in a trial if you take the system as a whole. I certainly think it is fair to discard the entire PISA England 2025 analysis and data – disregard them entirely. There are just too many red flags with both this data and the entire system of exams in England. I really do think league tables are likely to have messed up the integrity of the national exam data as a whole, quite apart from other problems league tables lead to, like teaching to the test and crowding out topics not on tests.
And what does the PISA 2025 England report itself say about all this? Well, the academics writing it up note in the main text on page 8 that
“A Non-Response Bias Analysis (NRBA) of the participating schools in England’s PISA 2025 sample suggested that the final set of participating schools was highly representative of the national distribution of schools in terms of several school-level characteristics. There was, however, some evidence that within those schools, non-responding pupils were slightly more likely to speak English as a first language, slightly more likely to have been eligible for free school meals at some point in the last 6 years of their education, and slightly more likely to receive a form of special educational needs support….. results should be interpreted cautiously, particularly when considering trends in performance over time and when making international comparison”
‘Slightly more likely’, hey? ‘Some evidence’, hey? And then the killer phrase: “results should be interpreted cautiously, particularly when considering trends in performance over time and when making international comparison.” There should be caution? When doing these comparisons is the whole point of the PISA studies and of course exactly what all the media fuss will focus on? It is like a bakery cautioning the clients not to eat the bread they bought. That, ladies and gentlemen, is how the derriere gets covered in academia.
Ok, if you deduced all this yourself, brilliant. You compare favourably to the writers of this PISA 2025 England report. No kidding: if you managed to do an analysis of the type above, your critical skills are fine in my opinion. To be able to read statistical tables in a wider context that is not mentioned in reports themselves (such as the possible link with league tables) is a niche ability, so well done if you came up with something like that yourself – no ALM education needed for you. If you could not do it, realise that you have the choice between dependency on others to tell you what is going on with data claims, or you can get a real education.
In conclusion
The story above reveals how you can game education statistics. That is particularly handy if you are a bureaucrat or school principal who has been criticized for decades that you are failing your students. You select the right schools, the right students, miss the right data, and you tell the poor performers not to show up for the tests that would expose what goes on. Bingo. You get to revel in adoration by politicians falling over themselves to be part of your success. Oh, but don’t forget to caution the audience somewhere in the reports you write! Then you are off the hook because, well, you did caution everybody, right?
Do I think there was a backroom in which this was mapped out? Possible, but more likely that the entire English education system has grown to have this relationship with data, because of those league tables. Individual teachers, schools, and even the department will just be getting by, doing the best for their neck of the woods, not rocking the boat and thus letting it all get worse over time. Who is really missing in action are the politicians.
Where is the cost to such data massaging? English students of course, since data ‘has shown’ that things are fine and do not need improvement but are going in the right direction. In stead of changing the system to deal with an exposed problem, the system will now build on its magnificent success, locking in bad policies even more.
Gaming statistics on a national scale is what the Soviet Union did. According to its own data, things were better every year …. till the hidden rot underneath made the whole thing collapse suddenly. Data massaging on a national scale reflects a system that is no longer learning.
Want to learn more about how to read data properly?
Join our free online information session on 16th September where we present our 1-year research apprenticeship program. For more information see the event site at Academia Libera Mentis.
The book “Minds that Dare” describes the journey of Academia Libera Mentis from both a teacher and student perspective.
Buy our new book Minds that Dare
1 I am here making the ball-park assumption that the distribution of grades is a step-function with the density of grades outside the 10th to 90th percentiles half of that in this central range. Its a crude approximation of the usual bell curve. I could make a better approximation if given access to the underlying micro-data.
