Mad, Bad and Dangerous Statistics

I don’t normally use this blog to bitch about other bloggers – the whole point about blogging is that it allows you to express your own opinions, however misguided they might be and it is up to your readers to decide if they are worth anything. So, this post is an exceptional one for me. Dr Eoin Clarke has a blog called The Green Benches where he produces a lot of apparently statistically based articles critiquing the government. It is pretty influential – the stats counter on the blog shows he’s had nearly a thousand times more views than me so lots of people must think he is saying worthwhile and interesting things. He doesn’t let my gentle and polite comments through his moderation process so here they are for you.

He has blogged today about how comprehensive schools have been proven to outperform independent schools by reference to data published by the Department for Education. In summary, this data (from page 21) shows that 80.7% of children at comprehensive schools get 5 GCSEs at A*-C compared with only 80.3% of children at independent schools. Dr Clarke uses this to support the following statement:

“Independent schools are free from state interference. They run their own show. These companies charge parents £1000s of pounds a year to enroll and educate little Timmy and Tabatha. But guess what? State Comprehensives under Local Education Authorities, burdened with all that state interference actually out-performed these independent schools in this year’s GCSEs. This is one of many reasons why the Free Schools agenda is mad, bad and dangerous.”
However, this is completely misleading. First, the data for independent schools, if you can be bothered to read the footnote to it states that it includes data for independent and non-maintained special schools. That is, schools for children whose special needs are such that they do not attend a mainstream school. Schools like the one that former Education Secretary Ruth Kelly sent her child to. The equivalent special schools in the maintained sector have their results separated out from those of comprehensives and achieved a pass rate of 1.3% getting the same measure of GCSEs trumpeted in the blog. Now, it may be that independent special schools select particularly intellectual special needs children, but somehow this seems unlikely and rather inconsistent with the concept of a special school. Even if the proportion of independently educated special needs children is low, at the very least this shows that Dr Clarke isn’t comparing like for like.
In any case, I hold no particular agenda for independent schools. Maybe some parents are wasting their money. Although they might also point to the figures for 5 GCSEs at A*-C including English and Maths which show a much higher proportion achieving this at independent schools to justify their decision.
The bigger issue is that the data does not really support the criticism of Free Schools. Free Schools are not allowed to be any more selective than comprehensives and are bound by the same admissions law(subject to the possibility of some exemptions, as yet unknown, as pointed out by Allan Beavis in his comment below), so their independence is different from that of the private schools. What private schools achieve is not particularly an indicator of what Free Schools might achieve. If, on the other hand, the hidden allegation is that Free Schools will inevitably become covertly selective, for example by being only located in areas where there are sufficient numbers of horrid pushy parents, another comparison that Dr Clarke might want to make is with the results for children in areas with selective schools.
That data shows that selective state grammar schools get 99.2% of their children achieving the measure Dr Clarke relies on. Big deal, they’re by nature selecting the very brightest children and dumping the rest into ghastly Secondary Moderns. Except, the data for Secondary Moderns shows that they get 77.8% achieving that same measure. Considering that they have by definition had the most academic children creamed off, and anecdotally at least, probably have a proportion of the ones from wealthier backgrounds who missed out on a grammar school place sent to independent schools instead, that looks like a tremendous result, a mere 3% behind their fortunate peers at comprehensives which include a wider selection of higher ability children. Overall, the percentage achieving Dr Clarke’s measure of success at GCSE seems to be nearer 90% in areas with selection rather than 81% in areas with comprehensives. If Free Schools are going to introduce selection into local areas, perhaps they are not nearly as mad, bad and dangerous as Dr Clarke believes.
I’d recommend anyone tempted by Dr Clarke’s blog to take all his statistical analyses with a very large pinch of salt.
(Updated to reflect the possibility of Free Schools’ Funding Agreements including exemptions from the admissions code)
Advertisements

First Day of the Happiest Days of Your Life

Today OMB started his first day in Reception class at school. The first of 14 years in compulsory education. Apart from a few worries he had about being good at making friends hopefully it went well. It is a nice school and the children seem happy and confident there. I’m a governor of the school and the presentations we’ve had from the Year 5 and 6 children on topics like bullying and school meals have been better researched and more fluently delivered than many I’ve seen from adults in the work environment.

I hope that he’ll be able to build on the friendships he already has with the handful of other 4 year olds going who live within a few doors of us and that he gets a chance to learn well. I’m not sure about how good our local secondary schools are, but a lot can change in 7 years. I hope that we don’t ever have to worry or think about whether his school experience is a good one that sets him up for the long and happy life that all should be able to achieve. I don’t want to have to agonise over school fees and whether it is right to go private, moving house to get a “better” catchment area or even setting up a Free School.

Fourteen years is a long time. I recently saw a photo of myself 14 years ago cutting my birthday cake, looking unbelievably fresh-faced and slim. It is weird to think that my mum in that photo was only a few years older than I am now. Weirder still to think that the changes OMB will go through in the next 14 years could be even more significant than my journey of the last 14 years through starting my career, buying my first home, marrying, moving 200 miles away and having OMB.

Good luck OMB!

Well-meaning policies not racist teaching to blame for low black attainment

Periodically the Guardian’s Comment Is Free website goes into meltdown over the plight of low educational achievement by black children. It has happened again today with an article headlined by a sub-editor angling for a job at a tabloid – “There’s no black in the baccalaureate” by Professor David Gillborn http://bit.ly/kIcSMg . The thesis put forward by the article is that black children are denied educational opportunity by a massive conspiracy of racist teachers throughout the country. Black children are not allowed to take the most valuable subjects at school and, of course, being the Guardian, this is only going to be made much worse by those dastardly Tories.

What the article doesn’t address is that the institutional racism it bemoans was meant to have been eradicated by now (it links to an article about a DfES report from 2006 concluding there was institutional racism in schools but that this could be addressed by 2010). That steps had apparently been taken to do so. That teaching is a profession filled with the core readership of the Guardian and education was for 13 years of the previous government subject to incredible amounts of policy direction aimed at making the system more inclusive.

As a school governor and someone who knows a fair number of teachers I find it difficult to believe that teachers are generally racist. If anything, my impression is that they are, or try very hard to be, professionally anti-racist. The general, if wrong-headed, view by the profession that mere membership of the BNP ought to be enough to disqualify a teacher from working as teachers is only part of this.

However, I do agree that education has developed institutional racism. Not of the caricature type sketched in the article and the linked report from 2006 but a much more serious and harmful one. Institutional racism introduced unintentionally while attempting to achieve the opposite by “progressive” policy-makers.

The cause of this unintended institutional racism is the use and abuse of the entire culture and apparatus of using ever more granular data to inform and drive education policy. League tables were the starting point but are the tip of this particular iceberg. Over more than ten years educational policy and its implementation in estimating outcomes for schools and individual pupils and setting targets for both has been based upon highly detailed data gathering and statistical analysis. Much of this has been conducted by the Fischer Family Trust (FFT) http://bit.ly/gtazg9 which works with every English local education authority (LEA) and provides them and their schools with highly sophisticated datasets and software for analysing pupil test score and assessment data. This is used to generate both estimates and targets for individual pupils and schools.

Well, you may ask, what’s wrong with basing education strategies on cold, hard, factual data and objective analysis? Surely that must be better than the bad old days (as described in the 2006 report) of letting teachers subjective prejudices rule the roost? The problem is that the almost irresistable temptation is to be ruled by the apparent objectivity of the data and clever statistical analysis tools. An outline of the data and analyses available can be found at http://bit.ly/izy0Wq .  Although the FFT itself cautions against unsophisticated use of its data and emphasises the importance of teacher’s judgements based on actual knowledge and experience of the children they teach, it is very difficult to go out on a limb against the numbers that are generated and the estimates of where individual children at a particular school “ought” to end up.

The FFT data is highly granular. It breaks down pupil attainment level probabilities by gender, whether they are born early or late in a school year, subject, postcode, school type economic class and ethnicity amongst other factors. Combined with actual data for individual pupils such as how they have historically done in tests, it produces remarkably accurate looking projections of what each child might achieve.

How does this explain the worrying phenomenon highlighted in the article of black children achieving the poorest grades? Simply, that the data of past low attainment by that group is then used to estimate and set targets in respect of members of that group in the future. Just as league table positioning incentivises schools to focus on getting those predicted D grades at GCSE up to a C, all that is needed for a school to appear to be doing a good job for black children is for it to get them to achieve at or slightly above the poor level historically achieved by that demographic. However, this applies also to other ethnicities, classes, etc – so the Chinese children who come out best by most statistical measures of attainment are targeted to meet or exceed those higher standards.

My son is about to start at Primary school this coming September. He will be starting in Reception with 4 other children from our street and the one behind it. They all come from broadly similar socio-economic backgrounds as all of our houses are similar, each has two co-habiting parents and so on. If they were of the same level of intelligence and attainment on starting school, the ordinary expectation would be, should be, that they ought to be treated the same and have similar expectations as to how they might progress through education. However, that is not how things work where tools like the FFT data analyses come into play. From day one of Reception Class onwards, the estimates for each and the targets that will be set for them to enable the school to determine whether it is succeeding will differ quite dramatically on the basis of their ethnicity and gender. From a personal perspective this isn’t so bad as fortuitously our son is in an ethnic category that is historically high attaining. It would be rather worse for his prospects were he to be black.

This is not because the teachers are racist. In the particular example I’m certain they are not (and they are confident and successful enough not to rely too heavily on FFT). It is because if the school he attended was not confident enough to discount things like FFT data it wouldn’t matter how hard he worked and how much he managed to achieve, he would always be predicted not to progress as much as he might had he had a different ethnicity even with all other factors being the same. When it comes to choosing subjects at GCSE, this would increase the probability that he might be discouraged from taking “hard” subjects that would contribute to gaining the EBacc qualification. A reasonable hypothesis might be that schools with lower Ofsted ratings and positions in league tables may be more inclined to game things to maximise their chances of showing improvement – if black or other under-achieving groups (such as white working class boys) are more likely to attend such schools the impact of using FFT data in an unsophisticated way will be even greater on those groups.

Individual teacher prejudice would compound this as a notional racist teacher may be less inclined to disagree with the low estimates turned out by the statistical analysis, but it would be a secondary factor; the counter-example would be a teacher who was prejudiced against Chinese children – it would be difficult for them to get away with setting targets below the high ones generated by the FFT data and teaching towards that.

Perhaps the answer is to let go of the cost and complexity of collecting, analysing and  interpreting all that data. To wean ourselves off our addiction to comparative study of spuriously objective quantitative descriptions. To become very wary of deducing from a description of how things are a prescription of how they ought to be.

Rather than castigating teachers as a group for their racism, institutionalised or otherwise, the answer would be to trust that on the whole teachers go into teaching to do the best they can for all the children they teach. Free schools might succeed by harnessing this even if the particular educational styles they adopt might be driven by subjective and debatable views of quality and success. So it might be less important whether they focus, like West London Free School, on the Classics than that they start from a perspective that all their pupils regardless of background can achieve a good number of creditable grades in “hard” subjects.

It would be a radical government that reacted to such success (if of course, Free Schools are successful) by providing these freedoms to schools more generally. Of course, it would be even more radical to do this without waiting to see how the Free Schools fare. I hope that this might be what is happening with the drive for many more schools to become Academies – a change that has received a fraction of the publicity of the 8 Free Schools coming in the first ripple. That wouldn’t have to mean that no data was collected about school and pupil achievement – there would still be an extent to which it was useful for some management purposes, just that it would revert to its proper subordinate position. For example, I imagine that it was helpful for my old school to know that while everyone in A and B set French got at least a C at GCSE/O level and that more than half the D set got more than a C, none of the C set got above a D – either there was a setting failure or failings in the C set’s teaching even though there were no official league tables at the time.

Finally, a small confession. I wouldn’t have achieved the EBac qualification myself as I was allowed to drop both History and Geography for GCSE. I haven’t done too badly educationally or professionally though so part of my radical proposal would be not to worry too much about proportions getting the EBacc as long as individual pupils studied things that were of value to them and achieved according to their ability and application. For some this might mean getting the EBacc, but not for all (as an aside my guess is that the EBacc would be much more valuable as a differentiator between children who leave school at 16 than those more academic children who go on to A levels, Higher Education and beyond).