Read on Twitter
A short thread to try to explain to non-UK people what seems to be happening with the A-level results in England&Wales.
Disclaimers:
1) There are lots of anecdotes going round
2) The situation is still not 100% clear
3) I have surely omitted some important stuff; please add.
/1 https://twitter.com/kIopptimistic/status/1293880274523217924
Disclaimers:
1) There are lots of anecdotes going round
2) The situation is still not 100% clear
3) I have surely omitted some important stuff; please add.
/1 https://twitter.com/kIopptimistic/status/1293880274523217924
First, some background. When final-year (age 17-18) school students in England and Wales apply to universities, they receive an offer of a place conditional on obtaining certain grades in their "A-level" (high school leaving) exams. /2
A-levels are quite specialised - most people take just 3 subjects (some take 4, e.g. including "General Studies"). Perhaps English/History/Music, or Maths/Physics/Chemistry. This specialisation is one reason why undergrad degrees in England & Wales take 3 years rather than 4. /3
Possible grades are A* ("A-star"), A, B, C, D, E, and U ("fail"). There has been some inflation over the years. To get into Oxford or Cambridge you need to pass a tough interview process and then get three A* grades. For a top 20 university you typically need A*A*A or A*AA. /4
When you apply to universities, offers come via a centralised system, and you choose two before your exams. So if you are offered A*A*A* by Cambridge you can also provisionally ("insurance") accept AAB from less prestigious university X. If you get A*AA then you can go to X. /5
Now of course this year there has been a lot of disruption to the system with COVID-19, and nobody (AFAIK) has actually taken any actual A-level exams. So the grades have been estimated. There are typically two inputs to the estimation process. /6
One is the results of the "mock" exams, which most students took at the start of this year. Typically these are administered by the school, perhaps using past A-level papers. They are intended to let the student get an idea of their strengths and weaknesses. /7
The other is an estimate by teachers of what grades a student is "capable of getting". These tend to be higher than what the students would, on average, have obtained, as people can have bad days, teachers are probably naturally a bit biased, etc. /8
So, those numbers have been put into a big model, the job of which is to predict individual grades for each student. This is where it gets messy. /9
The modellers don't have a lot of information about the individual students. They do, however, have a lot of information about the history of the education attainment of their town/suburb and/or individual school. So they've added that to the model. /10
On its face, this doesn't seem unreasonable. As anyone who models educational interventions will tell you, it's usually a good idea to add an effect for the school to the model, because of course a school that starts off with SES (etc) advantages will tend to do better. /11
But here's the problem. When the model encounters two students, both of whom scored A*AA in their mock exams and are predicted to get A*A*A by their teachers, the *only* basis for distinguishing between them is the other factors. /12
So if student 1 is at a top independent(*) school, and student 2 is at a below-average state(*) school, the model will almost inevitably predict that Student 2 will get lower grades. /13
(*) Don't even think about using the word "public" when discussing the English education system, unless you *really* know what you're doing. And even then, don't.
/14
/14
As a result (again, please see the disclaimers in tweet #1), we are seeing a whole lot of stories of people from lower-income families whose A*AA mocks and A*A*A predictions are coming out at ABD, and who are missing out on either their first choice or any university place. /15
If this analysis is correct (and of course it may very well not be), this is an embarrassing failure for the UK government, but it might also be a lesson for people who build statistical models and then use them to predict (or, in this case, prescribe) individual outcomes. /16
One of my "naïve outsider" hobby horses about the way we use models in psychology is that people like Gossett and Fisher, the inventors of our basic models, often used interchangeable participants that they had no reason to care about, e.g. bottles of Guinness or corn plants. /17
If your fertiliser causes 80% of plants to double in yield and the other 20% to die, you don't care. You've gained 60% in yield per hectare The central tendency is all you're interested in. /18
It's a bit different when your participants are humans. Primum non nocere. Even more so, I think, if you are a right-of-centre government and your model turns out to be systematically biased against disadvantaged people (again, see disclaimers in tweet #1). /19 /end
