[HN Gopher] Evidence of fraud in an influential field experiment...
___________________________________________________________________
Evidence of fraud in an influential field experiment about
dishonesty
Author : DominikPeters
Score : 111 points
Date : 2021-08-17 15:03 UTC (7 hours ago)
(HTM) web link (datacolada.org)
(TXT) w3m dump (datacolada.org)
| duxup wrote:
| What would the motivation be here? Just to get a paper out?
|
| Does the insurance company have any involvement / motivation?
|
| Outside someone at the insurance company who want an outcome of
| the paper to fit some goal of their own, hard to imagine the
| insurance company would "care" about the results enough to mess
| with the data. Although I'm open to the possibility that someone
| was just lazy and they wanted another dataset and so someone just
| fabricated it based on another dataset to just get it done with.
| milliondollar wrote:
| I can COMPLETELY believe it was a lazy analyst at an insurance
| company. Boss: where's the data that Bob wanted? er... Here!
| (Source: been inside many insurance companies.)
|
| Hanlon's (and Ocham's) Razor all the way here. Laziness /
| stupidity wins.
| bsder wrote:
| 100% agree. Especially if the data was going to be hard to
| collect.
|
| I have fabricated data to shut up my political chain more
| than once in my life. Why? Because they kept pestering me
| after being told that the data doesn't exist _yet_ but will
| exist naturally at some point in the future.
|
| So, I can fight with my management chain because some VP has
| "collect data about X" on his quarterly goals and simply
| won't take "No" for an answer. Or I can feed him crap data
| that he will most likely forget about. And if the data is
| actually important, the data will fix itself in <n> months
| when I collect it.
|
| Most probably, the data never gets looked at and I never
| waste the time collecting it. All good. I'm a wonderful team
| player that gets his job done. Probability: 95%
|
| Or, possibly, some intern comes to me in 18 months asking why
| my data seems to be ... off. Cool. Unbelievably, someone is
| really using that data. I give a "Hrm. I'll go look at that."
| prioritize the poor intern, collect the data and give them an
| attaboy for being so diligent. Intern is happy and his boss
| thinks he's extra diligent. Probability: 4%
|
| Or, if the data was _actually_ important, I collected it and
| resubmitted it myself at the first point we could
| realistically collect it because I wanted it for myself, too.
| Probability: 1%
|
| However, if that fabricated data somehow escaped the company
| and people depended upon it, yeah, egg yolk on the face all
| around, and I might get fired. Probability: 0% to a three
| digit engineering approximation.
| duxup wrote:
| Yeah the "fine here's your data, whatever" scenario with some
| disconnected guy who doesn't care, kinda believable.
| function_seven wrote:
| This is some damning info. Not only as evidence of outright
| fraud, but also of incompetence in that fraud. Uniformly
| generating something that's obviously not uniform, not even
| paying attention to fonts (!), and fabricating your result by
| adjusting not the collected "current" data, but rather the
| baseline data.
|
| Ariely's response[1] to this puts the fraud on the insurance
| company. He rightly notes that better data anomaly testing would
| have caught this, but I also wonder why the company "knew" which
| hypothesis to put their thumb on? And where did the impetus come
| from to double the number of records rather than leave N=6,744?
|
| [1]
| http://datacolada.org/storage_strong/DanBlogComment_Aug_16_2...
| frostburg wrote:
| I don't see how this could have happened without either
| collusion between the insurance company and the researchers or
| at least one of the researchers directly tampering with the
| data.
| function_seven wrote:
| Right? The insurance company doesn't have a hypothesis
| they're married to. You'd think they would really just want
| to know an actual technique to improving mileage honesty
| among their policyholders. It's the researcher who has the
| motive to show a significant finding. But I can also see
| Ariely's contact at the firm wanting to "help him out" by
| providing "good"(!) data.
|
| I'm going through the responses from the other 3 study
| authors, and I'm seeing a pattern in their replies. They're
| all--as gently and politely as possible--laying it at Dan
| Ariely's feet. (He does so as well in his own response; but
| there's a tiny whiff of skepticism--at least in my reading
| between lines--that he was blameless.) Mr. Bazerman's
| response seems the strongest in this regard.
|
| From Francesca Gino[1]:
|
| > _I start all my research collaborations from a place of
| trust and assume that all of my co-authors provide data
| collected with proper care and due diligence, and that they
| are presented with accuracy. In the case of Study 3, I was
| not involved in conversations with the insurance company that
| conducted the field experiment, nor in any of the steps of
| running the study or analyzing the data._
|
| From Max H. Bazerman[2]:
|
| > _The first time I saw the combined three-study paper was on
| February 23, 2011. On this initial reading, I thought I saw a
| problem with implausible data in Study 3. I raised the issue
| with a coauthor and was assured the data was accurate. I
| continued to ask questions because I was not convinced by the
| initial responses. When I eventually met another coauthor
| responsible for this portion of the work at a conference, I
| was provided more plausible explanations and felt more
| confidence in the underlying data. I would note that this
| coauthor quickly showed me the data file on a laptop; I did
| not nor did I have others examine the data more carefully._
|
| From Nina Mazar[3]:
|
| > _I want to make clear that I was not involved in conducting
| the field study, had no interactions with the insurance
| company, and don't know when, how, or by whom exactly the
| data was collected and entered. I have no knowledge of who
| fabricated the data._
|
| and
|
| > _This whole situation has reinforced the importance of
| having an explicit team contract, that clearly establishes
| roles, responsibilities, and processes_
|
| [1] http://datacolada.org/storage_strong/Gino-memo-data-
| colada-A...
|
| [2] http://datacolada.org/storage_strong/fraud.resonse.max_.8
| .13...
|
| [3] http://datacolada.org/storage_strong/20210816_NM-
| Response2Da...
| duxup wrote:
| Sounds like the classic case of a bunch of folks involved
| and the important thing is 'nobody's job' and so nobody
| does it.
|
| You see this in engineering failures when a bunch of
| companies or groups are involved to limited extents and
| everyone does their part, but nobody does something
| important because it wasn't defined who would do that.
| namelessoracle wrote:
| > The insurance company doesn't have a hypothesis they're
| married to
|
| While the insurance company as a whole didn't. There may
| have been someone inside the insurance company who wanted
| to justify spending funds to do this research and wanted a
| laurel on their cap about how they improved the accuracy of
| self reporting by X or Y value.
| rz2k wrote:
| Could it be one executive wanting get credit for his
| brilliant initiative by scientifically "proving" how
| effective it is?
|
| Using the uniform distribution makes it seem like it wasn't
| one of the researchers or anyone at the insurance company
| who has studied actuarial science.
| xondono wrote:
| While I see the possibility of someone at the insurance
| company being the "motivated party" to fudge with the data, I
| agree that it's difficult to see how one would do about it if
| one does not know what the researchers would look at.
|
| And if I was in charge of the study (or anyone with enough
| experience I think), that "test" should have been kept secret
| from the company.
|
| This to me implies at least blurry boundaries between the
| researcher (in this case Ariely) and someone at the company.
|
| I will also admit not to be very impartial to this, being to
| this day very unconvinced by Arielys research and especially
| his conclusions and the platform he has built around them.
| frostburg wrote:
| I've read all the individual statements by the authors and they
| don't even try to explain how this happened, they just vaguely
| bemoan their own trusting nature and deflect.
|
| I don't see how it would be plausible for the insurance company
| collecting the data to independently tamper with it in that
| specific way (and getting the typeface wrong) before passing it
| to the unsuspecting researchers.
|
| Oh, and of course they immediately taught the result to MBAs and
| executives. I wonder how long it'll take to filter out of the
| system.
| bediger4000 wrote:
| Does this count as an "easter egg"? I mean, fraud in an
| experiment about dishonesty is irony in its most ferrous form.
| prvc wrote:
| Not ironic, as I understand the concept.
| jjk166 wrote:
| > a state of affairs or an event that seems deliberately
| contrary to what one expects and is often amusing as a
| result.
|
| Dishonesty in a study on dishonesty is contrary to what I
| would expect and, at least in my opinion, amusing as a
| result.
| IncRnd wrote:
| It is a textbook example of situational irony for people who
| are on the outside of this situation, looking at how the
| results differ from what was expected.
| meowface wrote:
| It is extremely ironic, but I think the most ironic would be
| fraud in a research paper about research paper fraud.
| gameswithgo wrote:
| I can appreciate the irony at least.
| nn3 wrote:
| That was a long standing question of mine: Are people who build
| their careers around building experiments that mislead others (as
| a lot of psychological experiments do) more truthful in their
| papers? At least in this case the answer seems to be no.
| skynetv2 wrote:
| To get around the hug of death - https://archive.is/zayEm
| IncRnd wrote:
| In case you honestly can't reach the site, I've archived the page
| here - really.
|
| https://archive.is/zayEm
| superjan wrote:
| Small criticism: the histogram from the UK DoT uses varying size
| buckets, that makes the data look like a normal distribution. The
| histogram from the dataset is plotted with constant size buckets.
| It does not affect the conclusion though.
| [deleted]
| vincent-toups wrote:
| Whenever I see this stuff I wonder how much fraud is being
| perpetrated by people with enough statistical know how to make it
| hard to detect.
| derbOac wrote:
| From my own experiences this is the tip of the iceberg, but the
| majority isn't fraud per se, more like questionable research
| practices that cumulatively amount to something similar. So
| maybe not making up data, but fishing (either variables,
| people, or models) until you find the right combination. On top
| of _that_ are all the misuses of things, that aren 't really
| fishing, but rather use of methods that produce significant
| findings, but for reasons other than what is assumed.
| vlovich123 wrote:
| Look up Darrel Huff [1][2]. When there's enough money on the
| table, there's lots of funding (sincere or otherwise) that's
| used to try to establish a scientific counter narrative. Stuff
| like this is more run of the mill corruption than what you see
| with things like climate change where the underpinnings of the
| entire industrial economy is at stake (or milder things like
| Jule).
|
| [1] https://statmodeling.stat.columbia.edu/2012/04/27/how-to-
| mis... [2] https://en.wikipedia.org/wiki/Darrell_Huff
| garyfirestorm wrote:
| HN Hug of death
___________________________________________________________________
(page generated 2021-08-17 23:02 UTC)