Are the science questions measuring critical thinking or political correctness?
Throughout the education world, the PISA 2025 Science results have been presented as triumphs by the governments of countries high in the rankings, and discussed in hushed tones within countries low in the rankings. The key ‘performance table’ is below, taken straight from the OECD PISA 2025 results:
This table tells us that Chinese-like regions (which includes Singapore, Macao, Taipei, etc.) do really well in these tests.
Stories of Academia Libera Mentis is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.
Last week, I talked about data irregularities.1 This week, I want to dig into the content of the tests. What were the science questions? Who set them and what do the tests actually measure?
The main report, which is already 377 pages, does not tell one what these science results measure, apart from wonderfully sounding platitudes: ‘critical thinking’, ‘ability to apply concepts’, ‘practical science’, and so forth. The PISA team claims to do the exact same that we at ALM say we do, making me even more curious because the experience of the 30 academics or so of ALM has been that mainstream education stopped doing critical thinking a long time ago. But perhaps we ourselves are biased at ALM, or perhaps critical thinking is still alive in science education because science is less politically sensitive? Let us not judge too hastily and examine the questions.
The PISA 2025 Science questionnaire
I dug up the actual science questions for 2025. A few of those questions were very reasonable, such as the ones asking the students to work out how far away a storm was if one could see a flash of lightning and heard the sound of thunder a few seconds later. That kind of question is indeed applied science. Fine. There were also questions about the life-cycle of particular insects, which were also reasonable. One third (7 out of 21) were perfectly ok.
Yet, the majority of the 21 science questions in 2025 were problematic. Take the following question that I will dissect at some length.
Offshore wind turbine farm – question 3
Have a careful read and reflect on this question. Which of the four given answers would you select and what would your reasoning be?
Before looking at the answers, note that ‘Science’ is here about particular large-scale investments motivated by environmental language. That actually holds for most of the PISA Science questions, except the ones on storms and insects which I already mentioned. That is a somewhat narrow focus and of course does tend to the political: the student is already presumed to agree in this question that wind turbines make environmental sense, which my reading of the literature does not support: it takes huge amounts of energy to build these turbine farms, whilst those blades are an environmental nightmare once they are decommissioned because they are so tough and thus do not degrade. Anyhow, let us forgive the PISA for that framing.
What were my own deliberations on this question when I tried to answer it? For one, I reasoned that the construction of those wind turbines would often be done from the coast into the sea, meaning that one would have roads and machinery disrupting the beach and all the ‘lower’ zones: if one builds in zone D, one probably has to go through zone A, B, and C as well. An electricity cable originating from zone D would certainly have to go through zone A, B, and C to get onshore, deep below the seabed. Whilst it was not clear to me whether I was supposed to worry about environmental disruption during construction or over the whole life time of those wind turbines, I reasoned that the disruption in the operational phase would matter a lot more for overall impact than the temporary disruption during construction.
I then deduced that those wind turbines would help the growth of algae and molluscs and such. I reasoned that those wind turbines make a pretty good base for molluscs and algae and other life to grow on. Indeed, looking it up later, there is a scientific literature that calls those wind turbine parks ‘artificial reefs’.2 With those deliberations in mind, my answer was ‘B’, with as the argument that one is then not disturbing the birds on the coast whilst providing a better habitat for algae and such to feed other wildlife.
The supposed PISA answer? C. One got zero points for anything not C. And the main desired reasoning? Algae would be disturbed but fish would just swim away, just as mammals would, so there is supposedly no real environmental problem building in either zone C or D. Because building in zone C is cheaper, one was supposed to say C. The PISA manual thus says about the most desired explanations of students (and then the answers that would get some points):
Sigh, right? The answer ‘Zone B’ would have gotten me zero points. Note that lacking in any of these PISA-approved answers is the obvious point that construction in Zone C will involve disruption in Zone A and B as well. Wind turbines are not dropped into the sea by helicopter, you know! And those major electricity cables coming on shore need securing, which requires machinery and other disruptions. But anyhow.
Note how non-environmental these PISA-approved answers are: costs are for instance not equated to environmental damage. But how does one think costs arise? Via use of energy, machinery, and other inputs of course, which all have their own environmental impact. So, students here are supposed to think that costs are somehow environmentally neutral. That is childish and shallow. Note also the lack of deeper ecological reasoning here, such as the question of how these sea birds eat. They hunt for fish of course, so will fly to wherever the fish are (zone C). They might breed near the beach, but will have to go to zone C to eat, meaning that they will encounter those deadly wind turbines there. And so forth.
Reflecting on this, it seemed to me this question, and particularly the preferred answers, had nothing to do with science but was a strange kind of conformism test in which the students were supposed to be in sync with a very peculiar type of shallow thinking. ‘fish swim away so it’s ok to disturb them; algae do not swim so you should not do anything around them; costs are irrelevant’.
A truly scientific thinker should not give the answer this PISA question rewards This questions makes PISA 2025 Science look like a compliance meter, not a scientific proficiency meter.
Well, maybe I should not be so strident until I show other questions. The PISA 2025 Science test only had 21 questions, with about four on the construction of these wind turbines. Let me give one of the other environmental questions:
Energy source – question 3
Again, have a careful read and think about how you would answer to this question. Then read on.
As a technical question, this one is rather easy, but it does require one to believe something quite fantastical: that costs of the various energy sources are not merely per MWh, but can be shopped at any time of the day without any ‘fixed costs’. So for instance, the question description and the answers presume one can shop solar power for 5 units of money (‘Zeds’) whenever one wants in the day, though there are only so many units of solar power available at different times of the day (the availability is in the graph). It’s like going into a snackbar and ordering hamburgers, with limited availability of hamburgers over the day. There is a wind snack, a solar snackbar, and so forth.
That is a very strange assumption which violates the reality of energy provision: there is a huge fixed cost of solar panels and their installation, even if one never uses them at all! Hence if you want to be accurate, the first MWh one buys would cost a town millions of ZED, with the subsequent ones nearly for free in terms of additional costs. The dominance of fixed costs is the reality of energy provision choices, particularly wind, solar, and hydro, but that reality is assumed away in the description. Hence students are led to mis-understand the reality of costs via the framing of this question. The more informed have to grind their teeth and go along with it for the sake of answering.
There are also other hidden assumptions here that do not hold, such as that energy consumption is always the same every day (which is not true). That kind of assumption is more understandable and less disturbing: the question have to be somewhat simple, which is fair. On the other hand, the shown sources of energy are already a bit of a political choice. Why not nuclear energy, which is almost emission-free? Why not coal-powered, which is the form used a lot in reality because it is cheap? The mere choice of energy sources shown (wind, solar, hydro, and gas) is somewhat ‘politically correct’, though arguably not optimal for either the environment or costs.
Nevertheless, in the given fantasy world where one has four snackbars to shop energy from and one has to choose to have only two of those snackbars, the correct answer is obvious: one buys the hydro-electric first, and the wind as a supplement. To get that answer, the student has to puzzle a bit to see that with that combination indeed each hour of the day has enough energy provided to meet the fixed demand.
So at least in this question, I have no serious quibble with how you get the main points. I do have an issue with the follow-up open question here (“what is one risk to the town of the combination you have chosen”). It is essentially impossible to answer this follow-up honestly because one starts from a position so far away from reality that one is forced to also pick the ‘risks’ from the unknown fantasy land of what PISA scorers had in mind. My own top risk would be that the chosen assumptions are totally unrealistic and that one hence might have huge cost-overruns and not meet energy demand. That option would not get any points! Also, there is an irritating political framing in the question, and students do have to buy into a very strange view of energy costs that has almost no relation with reality. So the question is not applied science, but more applied science fiction. Or Science applied to Alice in Wonderland.
Energy source – question 5
A final question to discuss then, again on the town of Zed:
Here, the reader is invited to think of a carbon tax. It is fairly obvious that the student is supposed to think that gas is the form of energy that emits carbon, and that the student is not supposed to think of biogas or other emission-neutral forms of gas. The student is also clearly not supposed to think of the emissions involved in setting up a wind turbine park (those large concrete pylons which hold the turbine up don’t just appear out of nowhere, particularly not if you put them deep at sea!), or in the industrial processes that lead to solar panels, or in the decommissioning of wind and solar farms. Depending on the source of gas (could be from food waste!) and where those wind turbine parks are, more emissions are involved in wind turbines than in gas per unit of life-time energy.
Yet, the student is again clearly supposed not to think truly scientifically, but to have been pre-brainwashed into a very particular Alice in Wonderland view of environmental choices. It really is a political compliance test, though a rather obvious one in this case. Even though what I believe to be the most scientific answer (wind) is not the PISA answer, I too would have answered ‘gas’, knowing that would be the desired answer.
The second part of this question is more outlandish. To get full credit PISA requires that:
the student selects BOTH ‘to generate funding for research into clean energy innovations’ and ‘to encourage the use of energy sources that cause less environmental damage’
Talk about politically correct framing of these answers. Why not mention ‘spend on health’ as something a tax could be useful for? The environment is not the only worthy goal of government, you know.
However, apart from the obvious indoctrination aspect here, it is also not scientifically correct. The truly scientific student would NOT have answered either of those options for the reason that the one who actually has to pay the carbon tax in this hypothetical is the town government itself! After all, the previous questions made clear that the town of ZED has to decide on the energy sources to meet a fixed demand. It is not consumers paying the actual costs who decide (because then the demand would depend on the price and could not be shown as a fixed thing over the day!), but the town as a whole, taking energy consumption as fixed. So by the logic of these questions themselves, the town is in fact collecting a tax from itself, meaning that the only ‘correct’ response would have been no response at all. The town could have funded research anyway without this tax on itself, and if the town wanted to encourage less energy use, it should charge the town people the actual price per unit of energy they consume, which is very different policy that is not on the table.
Indeed, note a further idiocy in the question framing. The graphs show the costs per MWh for each of the energy sources and the carbon tax just adds to these costs because it is levied on the producers (which is the town itself). Within the logic of the previous question and this one, all the tax would do is make gas more expensive whilst having no effect on the other three. Since gas is already the most expensive one, it wouldn’t be used anyway, implying a carbon tax would have no effect on anything in this town, and indeed there would be no carbon tax revenue. But clearly, when the PISA speaks of ‘critical thinking’, consequential logic is not included.
One gets the drift: these questions are not politically neutral, with many being unscientific to boot, selecting for culturally-specific shallow thinking rather than critical broad thinking.
Who posed these questions?
Were the PISA science questions always like this? If not, who did the 2025 ones?
The PISA Science tests of a decade ago were much less infested by political correctness. When I look at the PISA 2015 Science questions, I see that they were about running in hot weather, asteroids, bird migration, solar radiation, and stuff like that. Much less politically correct, much more scientific, and indeed involving a higher level of reasoning required than in 2025.
So the PISA did not used to be like it is now. What happened? Who is setting the 2025 questions?
The OECD led the 2025 endeavour, but teamed up with a “design consortium led by the Australian Council for Educational Research (ACER)” to make the tests. ACER is indeed a politically correct body. Also, of course, it is Anglo-Saxon, which is inappropriate since that culture and its dominant political beliefs has infested the science questions, making them inappropriate for other cultures.
Who in particular designed this test? As usual, a committee, headed by Jonathan Osborne, prof emeritus at Stanford. We can gleam the character and political orientation of the man by reading an abstract from a 2023 published paper by him:3
In this paper, we argue that the current science curricula are failing to educate students to be competent outsiders to science.…. Today, misinformation abounds and much of it purports to be scientific. Very few conceptions of scientific literacy have considered how students can be prepared to evaluate the claims that abound on social media. Current conceptions of “scientific literacy,” we argue, are failing to articulate the competencies and knowledge required in today’s changed context…. we lay out what might be done within science education to address the specific phenomenon of misinformation.
Jonathan thus worries about misinformation and social media, and wants to “articulate the competencies and knowledge required in today’s changed context.” He seems to want to be the arbiter of what is useful and got his chance with PISA 2025. Going along obediently with Alice in Wonderland science seems to be what he means with “competencies and knowledge required in today’s changed context.”
Who else is responsible for these questions? On the committee, we find Doris Jorde (University of Oslo, Norway), Joe Krajcik (Michigan State University, United States), Yew Jin Lee (National Institute of Education, Singapore), Knut Neumann (Leibniz-Institute for Science and Mathematics Education, Germany), and Russell Tytler (Deakin University, Australia).
These are all very similar characters to the chair. Knut Neumann for instance, in a recent piece called Preparing Students for Living in an AI Infused World,4 complains about bad actors who question the supposedly settled science of climate change and vaccine safety.
We thus have the self-appointed guardians of settled science setting the PISA 2025 Science questions, coming up with tests of political obedience.
Let us judge whether these guardians of proper science are practicing what they preach. I alert the reader to a curious observation: if you look for the PISA 2025 test questions online, then unless you really go through the OECD website and pay a lot of attention there, you will often find the wrong questions! For instance, this is one of the supposed 2025 science questions journalists can find online:
I encountered this image when looking for the PISA 2025 reading questions. This question was not what I was looking for, but I was instantly surprised because I recognised this question. I had seen it just recently. It is from PISA 2015! I was amazed to find this one with the wrong ‘PISA 2025 label’ at the top. What? Why?
I first suspected that the picture was made by AI or some prankster. But no. It is actually PISA 2025 itself that generated it. To find it, you first have to go to https://pisa-framework.oecd.org/science-2025/ which gives background information to the PISA 2025 Science questions. There, if you scroll down to the bottom of that presentation, you find this screen:
Note the screen mentions that these seven categories are questions “of the kind that students might encounter.” A normal reader could think this means that these seven categories were the 2025 categories. Wrong. In actuality, these were the seven categories of the 2015 questionnaire.
What PISA says in this slide is not technically a lie, since in my judgment indeed 33% of PISA 2025 questions were of a similar ‘kind’ as these seven, but the claim is quite misleading because 67% of the actual 2025 questions were of a very different kind, and not just a lower quality kind. At the minimum, this screen is misdirecting. One wonders whether the journalists and academics who made it to this screen in order to figure out what the actual science questions were, will have investigated further.
If you then click on that button ‘7 Meteoroids and Craters’ on that screen, you indeed find:
Now, I see this screen, generated by the PISA team itself, as outright deception because of the false heading ‘PISA 2025’ above the question (together with those forward, backward buttons on the same row), making it seem as if this indeed was a PISA 2025 science question, which it is not. Talk about disinformation!
To be sure, on the OECD site with actual tests from 2015 you will find the pdf in which this question appears as follows:
Note the ‘PISA 2015’ top row? Stunning, right? PISA 2025 is in my opinion engaging in outright deception regarding the questions posed in PISA 2025 Science. It is diverting attention away from the politically charged questions that were actually in PISA 2025 by guiding readers to believe the questions asked were the much more sensible ones of 2015. It is deceptive. Hard to believe this was coincidence.
Who is responsible for this misinformation? The buck has to stop with the PISA 2025 scientific committee on this: they are ultimately responsible for what is wheeled out as the 2025 questions. Maybe the science team headed by Jonathan Osborne did not look at the website and are not aware of this deception, but they should know and certainly should act immediately to undo this deception. I will alert them myself.
Conclusions
The PISA science questions are best understood as a test of how brainwashed the students were when they came into the test room: true scientists would do poorly on this test. By my count, for about a third of the questions the ‘most scientific answer’ is counted as wrong by the PISA rules. A true scientists would thus score no higher than 2 out of 3.
If we take the first table in this article at face value, then Chinese teenagers are best at sniffing out desired non-scientific answers. I am not sure they should be too proud of that! Perhaps the countries doing worst on this test should be more proud of their students.
The cultural bias in these question is strong and it is thus not that mysterious anymore that Anglo-Saxon countries do so well on these science questions relative to their rankings in maths and reading: given that four out of the six question-makers were Anglo-Saxon (and the others strongly aligned with them), I am not surprised that students in those countries were more in tune with the political bias in these tests than students from other countries. For shame.
I think we should disregard the PISA science tests in their entirety from now on. PISA 2025 Science does not measure scientific literacy whilst actively undermining scientific thinking in favour of a shallow compliance with doctrine. It is a sad inditement on mainstream education that this is what it has come to at the international level. Serious education authorities should not take PISA Science seriously anymore.
