https://www.econlib.org/no-one-cared-about-my-spreadsheets/ [econlib-lo] The Library of Economics and Liberty * Articles * EconLog * EconTalk * Encyclopedia * Videos * Books * Guides EconLog | * Blog * Browse by Author * Browse by Topic * Search EconLog * RSS * Subscribe ECONLOG POST Feb 15 2022 Economic Methods Next > No One Cared About My Spreadsheets 21 By: Bryan Caplan No One Cared About My Spreadsheets Categories: Economic Methods By Bryan Caplan, Feb 15 2022 SHARE POST: The most painful part of writing The Case Against Education was calculating the return to education. I spent fifteen months working on the spreadsheets. I came up with the baseline case, did scores of "variations on a theme," noticed a small mistake or blind alley, then started over. Several programmer friends advised me to learn a new programming language like Python to do everything automatically, but I'm 98% sure that would have taken even longer - and introduced numerous additional errors into the results. I did plenty of programming in my youth, and I know my limitations. I took quality control very seriously. About half a dozen friends gave up whole days of their lives to sit next to me while I gave them a guided tour of the reasoning behind my number-crunching. Four years before the book's publication, I publicly released the spreadsheets, and asked the world to "embarrass me now" by finding errors in my work. If memory serves, one EconLog reader did find a minor mistake. When the book finally came out, I published final versions of all the spreadsheets underlying the book's return to education calculations. A one-to-one correspondence between what's in the book and what I shared with the world. Full transparency. Now guess what? Since the 2018 publication of The Case Against Education, precisely zero people have emailed me about those spreadsheets. The book enjoyed massive media attention. My results were ultra-contrarian: my preferred estimate of the Social Return to Education is negative for almost every demographic. I loudly used these results to call for massive cuts in education spending. Yet since the book's publication, no one has bothered to challenge my math. Not publicly. Not privately. No one cared about my spreadsheets. The upshot is that I probably could have saved a year of my life. I could have glossed over dozens of thorny issues. Taxes. Transfers. The effect of education on longevity. The effect of education on quality of life. The effect of education on crime. How unpleasant school is compared to work. Instead of reading multiple literatures to extract plausible parameters, I could have just eyeballed and stipulated for every tangential issue. Who would have called me on it? Don't get me wrong; The Case Against Education drew plenty of criticism. Almost none of it, however, was quantitative. Some critics appealed to common sense: "Education can't be anywhere near as wasteful as Caplan claims." Some critics called me a philistine: "Education isn't about making money; it's about becoming a whole person." Never mind that I wrote a whole chapter against this misinterpretation. A few critics bizarrely claimed that one recent paper had refuted my entire enterprise. But as far as I recall, zero critics ever checked my math. The most novel feature of my return to education calculations was that I tried to count everything that matters. I took the countless papers that start with the standard return estimates and tweak them with one novel complication. Then I merged all the tweaks that seemed convincing to me to get final policy-relevant numbers. If you wanted to use everything researchers know to craft optimal policy, that is precisely what you would do. In the end, however, I discovered that the true intellectual problem was not lack of supply, but lack of demand. Education researchers don't tweak standard return calculations to get the world closer to the truth. They tweak standard return calculations to get another publication - then move on with their lives. If the world handed out attention and tenure for synthesizing everything we know about the return to education, someone else would have done it long ago. It's hard to avoid a disheartening conclusion: Quantitative social science is barely relevant in the real world - and almost every social scientist covertly agrees. The complex math that researchers use is disposable. You deploy it to get a publication, then move on with your career. When it comes time to give policy advice, the math is AWOL. If you're lucky, researchers default to common sense. Otherwise, they go with their ideology and status-quo bias, using the latest prestigious papers as fig leaves. Empirical social science teaches us far more about the world than pure theory. Yet in practice, even empirical researchers barely care what empirical social science really has to teach. Next > --------------------------------------------------------------------- READER COMMENTS LEAVE A COMMENT * READ COMMENT POLICY Michael Rulle Feb 15 2022 at 9:26am Reply It does not matter that few people questioned your math----it matters that you seemingly went thru all the proper processes required to get it right. What matters is that it seems the large majority disagree with your conclusions---so they might believe the math is correct but somehow misses the point. Maybe college's greatest value is keeping 18-22 year olds locked up in a place where they can cause the least damage. Gates did not need it. My electrician did not need it. But they are the kind of people who could add value at 18. Me? I was a moron and filled with insecurity. I needed 4 years to get over it. Dylan Feb 15 2022 at 12:34pm Reply Me? I was a moron and filled with insecurity. I needed 4 years to get over it. I needed 20 (at least, still working on it) Ben Finn Feb 15 2022 at 1:54pm Reply I don't know if this is a well-known term but I've heard it called 'warehousing' teenagers. Or it may refer to a related concept, viz. keeping unruly teenagers adolescents in school doing nothing useful until they're mature enough to actually learn stuff. There's a similar concept I know of from a friend who works in Tanzania - sending teenage girls to school not so much to learn anything, because the schools are too abysmal for that, but just to keep them out of trouble (e.g. sexual abuse by older men). IronSig Feb 15 2022 at 5:40pm Reply Concentrating teenagers results in more dysfunctional teenage behavior, intended to selfishly impress and harass each other, than in placing them in the workplace. There, the damage they wreck will often come from correctable mistakes in communication and task assignment, and their chief motivations are adjusted away from (though maybe not entirely) showing off for each other to showing off for a paycheck and purposeful work. Even grunt work sweeping the shop floor or shredding old paperwork does more for preparing for the real world than preparing a diorama that will not be saved by your parents next to your baby pictures. MikeDC Feb 15 2022 at 10:20am Reply This is true well beyond empirics and into most of the "trappings" of reason. They're deployed in the service of folks underlying desires. Thus, aiming appeals to reason almost never works. Instead aim appeals to morality. Aaron Stewart Feb 15 2022 at 10:21am Reply Did you consider offering a bounty for finding major mistakes, or something similar? I don't know enough about the economics of publishing that sort of book to say where the bounty funds should come from, but I imagine they could come from the publisher, you, or from book sales. Then set a cut-off period after which the bounty money goes to you (or back to the publisher if they paid). Neutral but trusted party could adjudicate the merit of critics' claims. Maybe it only pays out if the mistake(s) identified change the final estimate of returns by more than some percentage. It would incentivize people to find mistakes, but also lend you credibility by showing you indisputably have skin in the game (reputational damage is always disputable). It would be great if you could also find a way to get potential claimants to publicly announce their intention to examine the data as they embark on it. That way, when responding to other critics or doing media appearances you can point to all the people that failed to find fault. Hell, given the opportunity, I would be willing to make a donation to the bounty fund if getting more money helps to lend credibility to the case. If it means the book actually winds up having any policy impact, I'd (in a perfect world) make the donation back in reduced taxes. Aaron Stewart Feb 15 2022 at 10:33am Reply Maybe a better way to set it up would be to consider the bounty to be just a necessary cost, and to choose a date (~5 years after publication) at which the bounty funds will be given to whoever made the best (most careful/impactful/etc) criticism. That will initially be a huge incentive for someone to make an initial, weak case against the conclusions. With that established, in order for someone else to snatch the bounty, they only need to make a slightly better case. With the other bounty-for-first-major-mistake model (as with the no-bounty model), there's a huge initial barrier to arguing against your case because there's a ton of work involved. With the bounty-for-biggest-mistake model, the initial barrier is very low, because their criticism only has to be better than no criticism at all. The idea here would be to encourage gradual commitment from critics in an escalating fashion. Aaron Stewart Feb 15 2022 at 10:36am Reply And then in 5 years when the bounty gets paid out, release a second edition correcting whatever mistakes were found! John Hall Feb 15 2022 at 12:03pm Reply Being thorough and making the spreadsheets public was good nonetheless. John hare Feb 15 2022 at 12:17pm Reply Most likely, you were considered to be reasoning from false premises. Once that conclusion is reached, spreadsheets become useless for persuasion. Charlie Feb 15 2022 at 12:33pm Reply I agree with this. Just because you consider yourself to have counted everything that matters does not mean that your readers are obliged to agree with you. The strongest criticism of your work is probably that you miss non-quantifiable returns to education, which makes your spreadsheet irrelevant. Alabamian Feb 15 2022 at 1:59pm Reply Bingo. Quantitative analysis is great and has its place. But don't confuse your map with the territory. Jamming an attribute into a scalar in a model is different than grappling with multi-dimensional reality in its full complexity. I feel like we need to get Russ Roberts over here to talk to Bryan. robc Feb 15 2022 at 2:08pm Reply Bryan has done at least 4 econtalk episodes, Feb 12, 2018 on this book in particular. Mark Z Feb 15 2022 at 4:07pm Reply I would say that's the weakest criticism of his book. It's trivially easy to postulate non-quantifiable benefits to something. The argument for spending measurable resources on unmeasurable (and thus unfalsifiable) benefits is almost always weak. Alan Goldhammer Feb 15 2022 at 1:38pm Reply Of course the math was correct as Excel, in my experience, never makes any mistakes. It religiously calculates what you input into it (I've done countless numbers of spreadsheets over the years for financial analysis). You are picking the wrong issue to write about. Some of us did read your book with a very critical eye and offered substantial comments on why various segments were either right or wrong (in my case there was much more wrong with the book than right). It's interesting that I could only find less than a handful of reviews of the book and only one in the mainstream press IIRC (The Washington Post which was pretty scathing "...it offers little more than dangerous, extravagant ideology masking as creative data analysis...") vince Feb 15 2022 at 1:42pm Reply One reason may be that the results were so, as you say, ultra-contrarian. A quick Google search shows median weekly earnings in Q3 of 2019 at $749 for high school diplomas but $1281 for Bachelor's degrees. The extra $28,000 per year is impressionable. Andrew_FL Feb 15 2022 at 2:32pm Reply Bryan doesn't dispute there are positive selfish returns to education. He found negative *social* returns to education. vince Feb 15 2022 at 3:53pm Reply That's why I said impressionable instead of impressive. I'm suggesting that many were content to assume the wage difference justified the value of education, and wouldn't bother looking further into the details. Andrew_FL Feb 15 2022 at 2:22pm Reply You didn't waste your time. If you had made your arguments without the spreadsheets-just guesstimating & eyeballing, you would've gotten quantitative criticism. A man who successfully deters burglars didn't waste his money on a security system just because it never got used. Bruce K. Britton Feb 15 2022 at 3:33pm Reply In your introductory slideshow, you don't say what the numbers are quantities of. Please let me know that. Peter Gerdes Feb 15 2022 at 4:13pm Reply What makes you think that no one checked your math? Presumably, what you know is that no one checked your math and was able to confidently identify an error. I mean, anyone who checked your math and didn't find any issues isn't going to make an issue of it. Absent finding major errors yourself, why would you come to the conclusion that no one checked your math rather than the conclusion that your math was sufficiently compelling as to not make a worthwhile target? Math like that is only going to get challenged *either* if you made mistakes or were sloppy (so someone can simply declare the methodology is sloppy or walk away) or if someone could generate a very different answer using the same general methodology but different plausible estimates. Maybe that's just not really possible in this case. (As an aside, I personally tend to agree with your conclusions about the direct benefits of education but are more pessimistic about our ability to avoid those disadvantages (eliminate college and you'd some other for-profit form of credentialism replace it). In particular, I fear that it wouldn't be possible to capture the social /fun/romantic benefits of sending young people all off to a common location to have fun together without the pretense it was about training even if, in theory, it should be possible to capture those advantages much much more cheaply.) LEAVE A COMMENT Cancel reply Your Name:required [ ] required Email Address:required, not displayed [ ] required, not displayed Website URL:optional [ ] optional Visual Text [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [SUBMIT COMMENT] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] This site uses Akismet to reduce spam. Learn how your comment data is processed. SHARE POST: Enter your email address to subscribe to our monthly newsletter: [ ] [ ] [Subscribe] COLLECTION: ECONOMIC METHODS The article you're reading is part of Econlib's Economic Methods collection. Explore other Economic Methods articles: * Feb 11 2022 Do Used Car Prices Vindicate Adam Smith? Pierre Lemieux * Feb 1 2022 Interpersonal Comparisons of Utility Pierre Lemieux * Jan 20 2022 I Win My Education Bet with David Henderson Bryan Caplan * Jan 13 2022 RCTs and the Status Quo: The Special Relationship Bryan Caplan The Library of Economics and Liberty Econlib The Library of Economics and Liberty Home page Articles EconLog EconTalk CEE Books Videos Guides Search About Econlib Contact Us Enter your email address to subscribe to our monthly newsletter: [ ] [ ] Subscribe Contact Us Privacy Policy Copyright @ 1999-2019 Liberty Fund, Inc. All Rights Reserved AdamSmithWorks Liberty Fund, Inc. Online Library of Liberty Law and Liberty Insert/edit link Close Enter the destination URL URL [ ] Link Text [ ] [ ] Open link in a new tab Or link to existing content Search [ ] No search term specified. Showing recent items. Search or use up and down arrow keys to select an item. Cancel [Add Link]