https://www.greaterwrong.com/posts/Yv9aj9bWD5H7aaDdy/how-my-school-gamed-the-stats ArchiveSequencesAbout [ ]Search Log InQuestionsEventsShortformAlignment ForumAF CommentsHomeFeatured AllTagsRecent Comments How my school gamed the stats Srdjan Miletic20 Feb 2021 19:23 UTC 63 points 11 commentsLW link EducationWorld ModelingCharter SchoolsGoodhart's Law [ ] Post permalinkLink without commentsLink without top nav barsLink without comments or top nav bars I was reading the Slate Star Codex review of Rule of the Smart. Part of it discusses charter vs state schools and the allegations of fraud of various kinds undermining charter schools record of better achievement. Reading it, I realized that I took for granted that public schools engage in systematic fraud in a variety of ways. I don't think this is something everyone understands, hence this post. I went to a state school in the UK. State schools are rated on a 1 - 4 scale from unsatisfactory to outstanding. My school was rated good, meaning a 3. A few memories which stand out. During my first week I saw one of the boys in my class who was 11 at the time held up against the wall in a corridor while a 16 year old put a shiv to his throat and robbed him. He handed over his wallet and keys. A year or two later and I remember seeing a small boy who struggled with depression held up by the throat against a locker and slapped in the face by a troublemaker from the same class in front of everyone just before we went in to the classroom. I remember classes which were filled start to finish with people shouting and talking. Neither of the first two events were common but they also weren't uncommon. No one was surprised to witness them. It's worth emphasizing again that my school was above average, in fact quite far above average, and in a middle class area. It's also worth noting that I was mostly in top ability streamed classes, meaning my classroom experience was likely far better than average. There were many ways in which the school and teachers gamed the system to boost their measured performance. One way was to do exams for students. I was on a bottom set language class for French. After two years I literally couldn't speak a single sentence in french and maybe knew 20 words in total. I still passed my exams. How? We did the tests in class. Often the teacher would go through them with us. Literally giving us the test and then going through each question on the whiteboard and telling us what to write. A different year and a different teacher, this time the teacher would sit next to us and write the answers down. Why sit next to us? It was the bottom set so people often wouldn't even bother to write down the answer if they were told it. This kind of thing was normal, so much so that I, and I think most people there, didn't realize anything unusual was happening. Another way schools game metrics is to cheat inspections. A major component of how schools are judged in the UK is through independent inspections carried out by an independent quasi-governmental organization called Ofsted. Now, you may imagine that these inspections would be unannounced, so as to best get a real image of how a school works. Not the case. They're scheduled well in advance. Before every inspection, a few things would happen in my school: * The worst troublemaker kids would be taken aside and put in a special room where inspectors wouldn't see them. Either that or they would just be told not to come into school at all on that day. * All of us were told in assembly that an inspection was coming and to be on our best behavior on that day. Often teachers would have conversations with less serious troublemakers and impress on them that they would behave on that day or face consequences afterwards. * Teachers would put a great deal more effort into their lesson plans than was normal. Classroom behavior management would also be far stricter. Because of these and other measures my school during an inspection was utterly different than my school on a normal day. On some level this isn't surprising. If teachers' promotions and management's jobs depend on good inspection results and inspections are easy to game, people will game them. Incentives drive behavior. But it's still sad. Another way the stats were gamed was by not recording bad behavior. When a school gives a detention or suspends/ expels a student, there's a record of it. This is especially true of suspensions, students being sent home or expulsions. The more of these you have, the worse you look as a school. The solution then is obvious, don't punish people or punish them in non-recorded ways. Again, in my school it was completely normal for students in lower sets to swear at the teacher, talk over them or disrupt the class for everyone else. It was normal for someone to be aggressive and abusive towards others and to face at most a 40 minute detention, but even getting a detention would be unusual. I realize that one data point is not enough to draw solid general conclusions. My own perception is that this kind of fraud wasn't specific to my school. My cousin went to a state school fairly nearby. He's 4 years younger than me. During one of my winters back from undergrad we discussed his school and his experiences mirrored mine. His exact words regarding inspections were "I learned 4 times more that day than any other day that year. It was amazing". I talked to a few British students at university, although specifically the not middle/ upper class ones who would have gone to public schools. They had gone to schools similar to mine in different parts of the country and their stories were similar and often worse. Two particularly funny examples from my friends' experiences stand out. A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window. My friend sitting in class noticed two boys making fun of him and then proceeded to get up in the middle of class while the teacher was talking, walk to their table, flip the table upwards to hit them in the face before going to sit down again when the teacher told him to. (Remember, my friend was a studious, sporty Asian kid and not a troublemaker. This kind of thing is normal in that environment). Comedic stories aside, my experiences in school, while not universal, seem fairly common in the UK and from what I've read of the statistics, bad US schools are far, far worse. I'm unsure what my point here is. I think I have two: * Charters may cook their books in various ways. In the UK, State schools do too. I would be surprised if it wasn't also the case in the US. * I think that I feel like a lot of commentators on places like SSC have fairly middle class experiences of fairly good schools and that bleeds into how their comparison between state vs charter schools. It's just good to remember that it's not those nice middle class schools that charters typically replace. Crossposted to my blog at https:// dissent.blog/ 2021/ 02/ 20/ how-my-school-gamed-the-stats/ Srdjan Miletic20 Feb 2021 19:23 UTC 63 points 11 commentsLW link EducationWorld ModelingCharter SchoolsGoodhart's Law [ ] Post permalinkLink without commentsLink without top nav barsLink without comments or top nav bars * Viliam 21 Feb 2021 1:14 UTC 13 points I am a former teacher. Not from UK--and as far as I know the situation in my country is not as bad (yet! growth mindset) -- but I definitely see some similar patterns. My favorite blog about teaching is Scenes From The Battleground (Teaching in British schools). The mechanism "if you punish students, it reflects badly on school, therefore schools tolerate things they in theory shouldn't" applies both to behavior and grades. Some students are shy, and when you tell them to stop doing something, they stop. But if they call your bluff... well, there is not really much you can do about it. The official advice provided by school administration is completely useless. If they disrupt the lessons, you can record their behavior, but unless there are about hundred records for given student per year, there are no consequences, so who cares. If they don't pay attention, and then fail to answer your question, and you give them a bad grade, they have a right to ask for independent re-examination. Which means that if they are smart, they can ignore you for most of the semester, ask for re-examination, cram for the exam, and get their grade fixed, maybe not to A, but B is still good enough for a semester of not working. Consequently, most teachers don't even bother to give grades other than As and Bs. Actually, giving bad grades usually reflects badly on the teacher (because you didn't teach them well, duh), so most teachers mostly give As. The relation between public and private schools is reversed in my country. Public schools are the boring mediocre, but most private ones are "pay to win". The teachers in private schools are paid slightly better than in public schools (like extra EUR100 a month), and in turn they are expected to turn a blind eye on everything, give everyone As, and generally not make trouble. The school inspections are just like you described (except for taking the worst kids away), everywhere. I taught shortly, so I only experienced one inspection as a teacher, and I was shocked how everything was completely different on that day. I didn't try to do anything differently, but kids were sufficiently instructed how to behave by the school administration. That was the only day when bad behavior could get them into actual trouble, and they understood the difference perfectly. Cheating on exams I only saw at the one private school where I taught briefly. At the end of high school, you take the Matura exam, where teachers from other schools (randomly assigned) observe the process: each student chooses a random question by lottery, then answers the question in front of a committee (consisting of teachers from that school, and the visiting teachers from other schools); the committee then collectively decides on the grade. On that private school, selected students were told in advance which question they would get, and some of them even got answers written on paper they had to memorize. Then the lottery was rigged, so each of them "randomly" selected the one and only question they prepared for. (There were plastic tokens with numbers of questions, randomly mixed face-down on the table while the teachers from other schools observed. Then one local teacher distracted the visitors by small talk, the other teacher peeked at the tokens and told students waiting in front of the classroom "choose the third token from right" etc.) + Srdjan Miletic 21 Feb 2021 2:34 UTC 3 points Parent I've read that blog too. It's pretty interesting. Do you have any other sources to recommend? Also, if you're willing to share which country did you teach in? o Viliam 21 Feb 2021 19:48 UTC 2 points Parent Answered in a private message, didn't want to dox myself publicly, it's a small country. * ejacob 21 Feb 2021 16:03 UTC 5 points A brief translation for any confused American readers: in UK English, "State School" refers to what we in the US call a "Public School." In UK English, "Public School" refers to what we in the US call a "Private School" (it is privately owned and/ or run but open to the 'public'). I think OP did a good job avoiding any possible confusion but this is a common enough error that it may be worth stating explicitly. * gjm 20 Feb 2021 22:48 UTC 5 points Just to provide a contrary datapoint: My daughter is in (UK state) secondary school right now, and nothing she has told us about her experiences there suggests that any of the awful behaviour you describe (of pupils, teachers, or school management) has been normal or anything like normal at her school at any time while she's been there. It's in a nice middle-class area with, e.g., quite a substantial fraction of parents having PhDs, and she's one of the most able students, all of which of course makes it more likely that the school has an easier time of it and that what she sees doesn't include the worst the school has to offer. But then you said that your school was in a nice middle-class area and you were mostly in top classes, so the two seem broadly comparable. My own experience of (UK state secondary) school was more like hers than yours, but as (1) that was back in the 1980s and (2) I was in a part of the country that had grammar schools and my school was an exceptionally strong one academically, that's of limited relevance to what most schools are like now. My wife's experience of (UK state secondary) school was also more like hers than yours; again, that was back in the 1980s, and again she was at an unusually strong school (not a grammar school, though as it happens its name had "Grammar School" in it for hysterical raisins). [UK state secondary school = US public high school] + Davidmanheim 21 Feb 2021 15:22 UTC 9 points Parent Note that only around 3% of UK residents have PhDs--so I strongly suspect that what you're calling "middle-class" is closer to the top 5% of the population, or what sociologists would say is the very upper part of the upper middle class. o gjm 21 Feb 2021 17:40 UTC 2 points Parent Well, the "substantial fraction" I have in mind isn't all that large. Certainly not more than 20% of pupils having >= one parent with a PhD ~= 10% of parents having PhDs, which would be ~3x the general population, suggesting that the population here might be comparable to the top ^ 1/[3] of the population as a whole. Kinda. Which is, as I say, certainly a nice middle-class area, but not much like taking the top 5% of the population. Class terminology is used differently by different people; the Wikipedia page for "upper middle class", which is probably reasonably representative, says that " [t]he upper middle class in Britain traditionally consists of the educated professionals who were born into higher-income backgrounds, such as legal professionals, executives, and surgeons"; interpreting "higher-income" fairly broadly, that might be 10% of the school's pupils. So no, not by any means "the very upper part of the upper middle class". (Also, we're near Cambridge, which means a lot of people with good academic qualifications; so in this population, good academic qualifications will be less evidence of e.g. wealth and social status than in the population at large.) + Srdjan Miletic 21 Feb 2021 2:33 UTC 2 points Parent My sample size is pretty small, limited to myself and a few people I knew, so I don't have a high degree of confidence that my experiences generalize across the UK. Hearing about experiences like your daughters shifts me somewhat towards thinking that state schools are okayish on average. Still, I find it hard to convert the various good and bad stories I hear into any kind of high confidence conclusion without hard data, which I haven't managed to find much of. * Bucky 21 Feb 2021 8:42 UTC 4 points I find it interesting that company audits (that I've experienced anyway) suffer from the same problem as ofstead inspections. It is perhaps worth noting that Ofstead inspections are nowadays done with a day advance warning and can be done with no warning. + TheMajor 21 Feb 2021 11:31 UTC 1 point Parent What do you mean 'problem'? Everybody involved wants the inspection to go well, the correlation between the outcome of the inspection and the quality of the school/ firm's books is incidental at best. * JMatthews 21 Feb 2021 21:16 UTC 1 point I think your experience is very typical, if not a wee bit better, than most uk school kids. I am in my third year teaching in a nice middle class area yet there are still huge behaviour issues at my school. One kid in year 9 burnt down the toilets last term...! I hear many similar reports from my colleagues in other schools across the area. I would make some disagreements on the inspections. Schools now get far less notice. Its just the night before were told ofsted will come. Also, if a school was found to be taking trouble makers out i imagine they would find themselves in special measures very quickly. Also, I'm not sure how a school could game assessments which have external examiners? With coursework that could be an issue but thats all but eliminated now. Back to top